Modulating a size of pixels displayed to a user for augmented or virtual reality

ABSTRACT

Configurations are disclosed for presenting virtual reality and augmented reality experiences to users. The system may comprise an image-generating source to provide one or more frames of image data in a time-sequential manner, a light modulator configured to transmit light associated with the one or more frames of image data, a substrate to direct image information to a user&#39;s eye, wherein the substrate houses a plurality of reflectors, a first reflector of the plurality of reflectors to reflect transmitted light associated with a first frame of image data at a first angle to the user&#39;s eye, and a second reflector to reflect transmitted light associated with a second frame of the image data at a second angle to the user&#39;s eye.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of pending U.S. patent applicationSer. No. 14/555,585, filed Nov. 27, 2014, entitled “VIRTUAL ANDAUGMENTED REALITY SYSTEMS AND METHODS”, which claims priority from U.S.Provisional Application Ser. No. 61/909,774, filed Nov. 27, 2013,entitled “VIRTUAL AND AUGMENTED REALITY SYSTEMS AND METHODS”. Thecontents of the aforementioned applications are hereby expresslyincorporated by reference into the present application their entireties.

FIELD OF THE INVENTION

The present disclosure relates to virtual reality and augmented realityimaging and visualization systems.

BACKGROUND

Modern computing and display technologies have facilitated thedevelopment of systems for so called “virtual reality” or “augmentedreality” experiences, wherein digitally reproduced images or portionsthereof are presented to a user in a manner wherein they seem to be, ormay be perceived as, real. A virtual reality, or “VR”, scenariotypically involves presentation of digital or virtual image informationwithout transparency to other actual real-world visual input; anaugmented reality, or “AR”, scenario typically involves presentation ofdigital or virtual image information as an augmentation to visualizationof the actual world around the user. For example, referring to FIG. 1,an augmented reality scene (4) is depicted wherein a user of an ARtechnology sees a real-world park-like setting (6) featuring people,trees, buildings in the background, and a concrete platform (1120). Inaddition to these items, the user of the AR technology also perceivesthat he “sees” a robot statue (1110) standing upon the real-worldplatform (1120), and a cartoon-like avatar character (2) flying by whichseems to be a personification of a bumble bee, even though theseelements (2, 1110) do not exist in the real world. As it turns out, thehuman visual perception system is very complex, and producing a VR or ARtechnology that facilitates a comfortable, natural-feeling, richpresentation of virtual image elements amongst other virtual orreal-world imagery elements is challenging.

Referring to FIG. 2A, stereoscopic wearable glasses (8) typeconfigurations have been developed which generally feature two displays(10, 12) that are configured to display images with slightly differentelement presentation such that a three-dimensional perspective isperceived by the human visual system. Such configurations have beenfound to be uncomfortable for many users due to a mismatch betweenvergence and accommodation which must be overcome to perceive the imagesin three dimensions; indeed, some users are not able to toleratestereoscopic configurations. FIG. 2B shows another stereoscopic wearableglasses (14) type configuration featuring two forward-oriented cameras(16, 18) configured to capture images for an augmented realitypresentation to the user through stereoscopic displays. The position ofthe cameras (16, 18) and displays generally blocks the natural field ofview of the user when the glasses (14) are mounted on the user's head.

Referring to FIG. 2C, an augmented reality configuration (20) is shownwhich features a visualization module (26) coupled to a glasses frame(24) which also holds conventional glasses lenses (22). The user is ableto see an at least partially unobstructed view of the real world withsuch a system, and has a small display (28) with which digital imagerymay be presented in an AR configuration to one eye—for a monocular ARpresentation. FIG. 2D features a configuration wherein a visualizationmodule (32) may be coupled to a hat or helmet (30) and configured topresent monocular augmented digital imagery to a user through a smalldisplay (34). FIG. 2E illustrates another similar configuration whereina frame (36) couple-able to a user's head in a manner similar to aneyeglasses coupling so that a visualization module (38) may be utilizedto capture images and also present monocular augmented digital imageryto a user through a small display (40). Such a configuration isavailable, for example, from Google, Inc., of Mountain View, Calif.under the trade name GoogleGlass®. None of these configurations isoptimally suited for presenting a rich, binocular, three-dimensionalaugmented reality experience in a manner that will be comfortable andmaximally useful to the user, in part because prior systems fail toaddress some of the fundamental aspects of the human perception system,including the photoreceptors of the retina and their interoperation withthe brain to produce the perception of visualization to the user.

Referring to FIG. 3, a simplified cross-sectional view of a human eye isdepicted featuring a cornea (42), iris (44), lens—or “crystalline lens”(46), sclera (48), choroid layer (50), macula (52), retina (54), andoptic nerve pathway (56) to the brain. The macula is the center of theretina, which is utilized to see moderate detail; at the center of themacula is a portion of the retina that is referred to as the “fovea”,which is utilized for seeing the finest details, and which contains morephotoreceptors (approximately 120 cones per visual degree) than anyother portion of the retina. The human visual system is not a passivesensor type of system; it is configured to actively scan theenvironment. In a manner somewhat akin to use of a flatbed scanner tocapture an image, or use of a finger to read Braille from a paper, thephotoreceptors of the eye fire in response to changes in stimulation,rather than constantly responding to a constant state of stimulation.Thus motion is required to present photoreceptor information to thebrain (as is motion of the linear scanner array across a piece of paperin a flatbed scanner, or motion of a finger across a word of Brailleimprinted into a paper). Indeed, experiments with substances such ascobra venom, which has been utilized to paralyze the muscles of the eye,have shown that a human subject will experience blindness if positionedwith his eyes open, viewing a static scene with venom-induced paralysisof the eyes. In other words, without changes in stimulation, thephotoreceptors do not provide input to the brain and blindness isexperienced. It is believed that this is at least one reason that theeyes of normal humans have been observed to move back and forth, ordither, in side-to-side motion in what are called “microsaccades”.

As noted above, the fovea of the retina contains the greatest density ofphotoreceptors, and while humans typically have the perception that theyhave high-resolution visualization capabilities throughout their fieldof view, they generally actually have only a small high-resolutioncenter that they are mechanically sweeping around a lot, along with apersistent memory of the high-resolution information recently capturedwith the fovea. In a somewhat similar manner, the focal distance controlmechanism of the eye (ciliary muscles operatively coupled to thecrystalline lens in a manner wherein ciliary relaxation causes tautciliary connective fibers to flatten out the lens for more distant focallengths; ciliary contraction causes loose ciliary connective fibers,which allow the lens to assume a more rounded geometry for more close-infocal lengths) dithers back and forth by approximately ¼ to ½ diopter tocyclically induce a small amount of what is called “dioptric blur” onboth the close side and far side of the targeted focal length; this isutilized by the accommodation control circuits of the brain as cyclicalnegative feedback that helps to constantly correct course and keep theretinal image of a fixated object approximately in focus.

The visualization center of the brain also gains valuable perceptioninformation from the motion of both eyes and components thereof relativeto each other. Vergence movements (i.e., rolling movements of the pupilstoward or away from each other to converge the lines of sight of theeyes to fixate upon an object) of the two eyes relative to each otherare closely associated with focusing (or “accommodation”) of the lensesof the eyes. Under normal conditions, changing the focus of the lensesof the eyes, or accommodating the eyes, to focus upon an object at adifferent distance will automatically cause a matching change invergence to the same distance, under a relationship known as the“accommodation-vergence reflex.” Likewise, a change in vergence willtrigger a matching change in accommodation, under normal conditions.Working against this reflex, as do most conventional stereoscopic AR orVR configurations, is known to produce eye fatigue, headaches, or otherforms of discomfort in users.

Movement of the head, which houses the eyes, also has a key impact uponvisualization of objects. Humans move their heads to visualize the worldaround them; they often are in a fairly constant state of repositioningand reorienting the head relative to an object of interest. Further,most people prefer to move their heads when their eye gaze needs to movemore than about 20 degrees off center to focus on a particular object(i.e., people do not typically like to look at things “from the cornerof the eye”). Humans also typically scan or move their heads in relationto sounds—to improve audio signal capture and utilize the geometry ofthe ears relative to the head. The human visual system gains powerfuldepth cues from what is called “head motion parallax”, which is relatedto the relative motion of objects at different distances as a functionof head motion and eye vergence distance (i.e., if a person moves hishead from side to side and maintains fixation on an object, itemsfarther out from that object will move in the same direction as thehead; items in front of that object will move opposite the head motion;these are very salient cues for where things are spatially in theenvironment relative to the person—perhaps as powerful as stereopsis).Head motion also is utilized to look around objects, of course.

Further, head and eye motion are coordinated with something called the“vestibulo-ocular reflex”, which stabilizes image information relativeto the retina during head rotations, thus keeping the object imageinformation approximately centered on the retina. In response to a headrotation, the eyes are reflexively and proportionately rotated in theopposite direction to maintain stable fixation on an object. As a resultof this compensatory relationship, many humans can read a book whileshaking their head back and forth (interestingly, if the book is pannedback and forth at the same speed with the head approximately stationary,the same generally is not true—the person is not likely to be able toread the moving book; the vestibulo-ocular reflex is one of head and eyemotion coordination, generally not developed for hand motion). Thisparadigm may be important for augmented reality systems, because headmotions of the user may be associated relatively directly with eyemotions, and the system preferably will be ready to work with thisrelationship.

Indeed, given these various relationships, when placing digital content(e.g., 3-D content such as a virtual chandelier object presented toaugment a real-world view of a room; or 2-D content such as aplanar/flat virtual oil painting object presented to augment areal-world view of a room), design choices may be made to controlbehavior of the objects. For example, the 2-D oil painting object may behead-centric, in which case the object moves around along with theuser's head (e.g., as in a GoogleGlass approach); or the object may beworld-centric, in which case it may be presented as though it is part ofthe real world coordinate system, so that the user may move his head oreyes without moving the position of the object relative to the realworld.

Thus when placing virtual content into the augmented reality worldpresented with an augmented reality system, whether the object should bepresented as world centric (i.e., the virtual object stays in positionin the real world so that the user may move his body, head, eyes aroundit without changing its position relative to the real world objectssurrounding it, such as a real world wall); body, or torso, centric, inwhich case a virtual element may be fixed relative to the user's torso,so that the user can move his head or eyes without moving the object,but that is slaved to torso movements; head centric, in which case thedisplayed object (and/or display itself) may be moved along with headmovements, as described above in reference to GoogleGlass; or eyecentric, as in a “foveated display” configuration, as is describedbelow, wherein content is slewed around as a function of what the eyeposition is.

With world-centric configurations, it may be desirable to have inputssuch as accurate head pose measurement, accurate representation and/ormeasurement of real world objects and geometries around the user,low-latency dynamic rendering in the augmented reality display as afunction of head pose, and a generally low-latency display.

The systems and techniques described herein are configured to work withthe visual configuration of the typical human to address thesechallenges.

SUMMARY

Embodiments of the present invention are directed to devices, systemsand methods for facilitating virtual reality and/or augmented realityinteraction for one or more users. In one aspect, a system fordisplaying virtual content is disclosed.

In one or more embodiment, the system comprises an image-generatingsource to provide one or more frames of image data in a time-sequentialmanner, a light modulator configured to transmit light associated withthe one or more frames of image data, a substrate to direct imageinformation to a user's eye, wherein the substrate houses a plurality ofreflectors, a first reflector of the plurality of reflectors to reflectlight associated with a first frame of image data at a first angle tothe user's eye, and a second reflector of the plurality of reflectors toreflect light associated with a second frame of image data at a secondangle to the user's eye.

In another embodiment, a system for displaying virtual content comprisesan image-generating source to provide one or more frames of image datain a time-sequential manner, a display assembly to project light raysassociated with the one or more frames of image data, the displayassembly comprises a first display element corresponding to a firstframe-rate and a first bit depth, and a second display elementcorresponding to a second frame-rate and a second bit depth, and avariable focus element (VFE) configurable to vary a focus of theprojected light and transmit the light to the user's eye.

In yet another embodiment, a system for displaying virtual contentcomprises an array of optical fibers to transmit light beams associatedwith an image to be presented to a user, and a lens coupled to the arrayof the optical fibers to deflect a plurality of light beams output bythe array of optical fibers through a single nodal point, wherein thelens is physically attached to the optical fibers such that a movementof the optical fiber causes the lens to move, and wherein the singlenodal point is scanned.

In another embodiment, a virtual reality display system comprises aplurality of optical fibers to generate light beams associated with oneor more images to be presented to a user, and a plurality of phasemodulators coupled to the plurality of optical fibers to modulate thelight beams, wherein the plurality of phase modulators modulate thelight in a manner that affects a wavefront generated as a result of theplurality of light beams.

In one embodiment, a system for displaying virtual content to a usercomprises a light projection system to project light associated with oneor more frames of image data to a user's eyes, the light project systemconfigured to project light corresponding to a plurality of pixelsassociated with the image data and a processor to modulate a size of theplurality of pixels displayed to the user.

In one embodiment, a system of displaying virtual content to a user,comprises an image-generating source to provide one or more frames ofimage data, a multicore assembly comprising a plurality of multicorefibers to project light associated with the one or more frames of imagedata, a multicore fiber of the plurality of multicore fibers emittinglight in a wavefront, such that the multicore assembly produces anaggregate wavefront of the projected light, and a phase modulator toinduce phase delays between the multicore fibers in a manner such thatthe aggregate wavefront emitted by the multicore assembly is varied,thereby varying a focal distance at which the user perceives the one ormore frames of image data.

In another embodiment, a system for displaying virtual content to a usercomprises an array of microprojectors to project light beams associatedwith one or more frames of image data to be presented to the user,wherein the microprojector is configurable to be movable relative to oneor more microprojectors of the array of the microprojectors, a frame tohouse the array of microprojectors, a processor operatively coupled tothe one or more microprojectors of the array of microprojectors tocontrol one or more light beams transmitted from the one or moreprojectors in a manner such that the one or more light beams aremodulated as a function of a position of the one or more microprojectorsrelative to the array of microprojectors, thereby enabling delivery of alightfield image to the user.

Additional and other objects, features, and advantages of the inventionare described in the detail description, figures and claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a user's view of augmented reality (AR) through awearable AR user device, in one illustrated embodiment.

FIGS. 2A-2E illustrates various embodiments of wearable AR devices.

FIG. 3 illustrates a cross-sectional view of the human eye, in oneillustrated embodiment.

FIGS. 4A-4D illustrate one or more embodiments of various internalprocessing components of the wearable AR device.

FIGS. 5A-5H illustrate embodiments of transmitting focused light to auser through a transmissive beamsplitter substrate.

FIGS. 6A and 6B illustrate embodiments of coupling a lens element withthe transmissive beamsplitter substrate of FIGS. 5A-5H.

FIGS. 7A and 7B illustrate embodiments of using one or more waveguidesto transmit light to a user.

FIGS. 8A-8Q illustrate embodiments of a diffractive optical element(DOE).

FIGS. 9A and 9B illustrate a wavefront produced from a light projector,according to one illustrated embodiment.

FIG. 10 illustrates an embodiment of a stacked configuration of multipletransmissive beamsplitter substrate coupled with optical elements,according to one illustrated embodiment.

FIGS. 11A-11C illustrate a set of beamlets projected into a user'spupil, according to the illustrated embodiments.

FIGS. 12A and 12B illustrate configurations of an array ofmicroprojectors, according to the illustrated embodiments.

FIGS. 13A-13M illustrate embodiments of coupling microprojectors withoptical elements, according to the illustrated embodiments.

FIGS. 14A-14F illustrate embodiments of spatial light modulators coupledwith optical elements, according to the illustrated embodiments.

FIGS. 15A-15C illustrate the use of a wedge type waveguides along with aplurality of light sources, according to the illustrated embodiments.

FIGS. 16A-16O illustrate embodiments of coupling optical elements tooptical fibers, according to the illustrated embodiments.

FIG. 17 illustrates a notch filter, according to one illustratedembodiment.

FIG. 18 illustrates a spiral pattern of a fiber scanning display,according to one illustrated embodiment.

FIGS. 19A-19N illustrate occlusion effects in presenting a darkfield toa user, according to the illustrated embodiments.

FIGS. 20A-20O illustrate embodiments of various waveguide assemblies,according to the illustrated embodiments.

FIGS. 21A-21N illustrate various configurations of DOEs coupled to otheroptical elements, according to the illustrated embodiments.

FIGS. 22A-22Y illustrate various configurations of freeform optics,according to the illustrated embodiments.

DETAILED DESCRIPTION

Referring to FIGS. 4A-4D, some general componentry options areillustrated. In the portions of the detailed description which followthe discussion of FIGS. 4A-4D, various systems, subsystems, andcomponents are presented for addressing the objectives of providing ahigh-quality, comfortably-perceived display system for human VR and/orAR.

As shown in FIG. 4A, an AR system user (60) is depicted wearing a frame(64) structure coupled to a display system (62) positioned in front ofthe eyes of the user. A speaker (66) is coupled to the frame (64) in thedepicted configuration and positioned adjacent the ear canal of the user(in one embodiment, another speaker, not shown, is positioned adjacentthe other ear canal of the user to provide for stereo/shapeable soundcontrol). The display (62) is operatively coupled (68), such as by awired lead or wireless connectivity, to a local processing and datamodule (70) which may be mounted in a variety of configurations, such asfixedly attached to the frame (64), fixedly attached to a helmet or hat(80) as shown in the embodiment of FIG. 4B, embedded in headphones,removably attached to the torso (82) of the user (60) in abackpack-style configuration as shown in the embodiment of FIG. 4C, orremovably attached to the hip (84) of the user (60) in a belt-couplingstyle configuration as shown in the embodiment of FIG. 4D.

The local processing and data module (70) may comprise a power-efficientprocessor or controller, as well as digital memory, such as flashmemory, both of which may be utilized to assist in the processing,caching, and storage of data a) captured from sensors which may beoperatively coupled to the frame (64), such as image capture devices(such as cameras), microphones, inertial measurement units,accelerometers, compasses, GPS units, radio devices, and/or gyros;and/or b) acquired and/or processed using the remote processing module(72) and/or remote data repository (74), possibly for passage to thedisplay (62) after such processing or retrieval. The local processingand data module (70) may be operatively coupled (76, 78), such as via awired or wireless communication links, to the remote processing module(72) and remote data repository (74) such that these remote modules (72,74) are operatively coupled to each other and available as resources tothe local processing and data module (70).

In one embodiment, the remote processing module (72) may comprise one ormore relatively powerful processors or controllers configured to analyzeand process data and/or image information. In one embodiment, the remotedata repository (74) may comprise a relatively large-scale digital datastorage facility, which may be available through the internet or othernetworking configuration in a “cloud” resource configuration. In oneembodiment, all data is stored and all computation is performed in thelocal processing and data module, allowing fully autonomous use from anyremote modules.

Referring to FIGS. 5A through 22Y, various display configurations arepresented that are designed to present the human eyes with photon-basedradiation patterns that can be comfortably perceived as augmentations tophysical reality, with high-levels of image quality andthree-dimensional perception, as well as being capable of presentingtwo-dimensional content.

Referring to FIG. 5A, in a simplified example, a transmissivebeamsplitter substrate (104) with a 45-degree reflecting surface (102)directs incoming radiation (106), which may be output from a lens (notshown), through the pupil (45) of the eye (58) and to the retina (54).The field of view for such a system is limited by the geometry of thebeamsplitter (104). To accommodate the desire to have comfortableviewing with minimal hardware, in one embodiment, a larger field of viewcan be created by aggregating the outputs/reflections of variousdifferent reflective and/or diffractive surfaces and using, e.g., aframe-sequential configuration wherein eye (58) is presented with asequence of frames at high frequency that provides the perception of asingle coherent scene. As an alternative to, or in addition to,presenting different image data via different reflectors in atime-sequential fashion, the reflectors may separate content by othermeans, such as polarization selectivity or wavelength selectivity. Inaddition to being capable of relaying two-dimensional images, thereflectors can relay the three-dimensional wavefronts associated withtrue-three-dimensional viewing of actual physical objects.

Referring to FIG. 5B, a substrate (108) comprising a plurality ofreflectors at a plurality of angles (110) is shown, with each reflectoractively reflecting in the depicted configuration for illustrativepurposes. The reflectors may be switchable elements to facilitatetemporal selectivity. In one embodiment, the reflective surfaces wouldintentionally be sequentially activated with frame-sequential inputinformation (106), in which each reflective surface presents a narrowfield of view sub-image which is tiled with other narrow field of viewsub-images presented by the other reflective surfaces to form acomposite wide field of view image. For example, referring to FIGS. 5C,5D, and 5E, surface (110), about in the middle of substrate (108), isswitched “on” to a reflecting state, such that it reflects incomingimage information (106) to present a relatively narrow field of viewsub-image in the middle of a larger field of view, while the otherpotential reflective surfaces are in a transmissive state.

Referring to FIG. 5C, incoming image information (106) coming from theright of the narrow field of view sub-image (as shown by the angle ofincoming beams 106 relative to the substrate 108 input interface 112,and the resultant angle at which they exit the substrate 108) isreflected toward the eye (58) from reflective surface (110). FIG. 5Dillustrates the same reflector (110) active, with image informationcoming from the middle of the narrow field of view sub-image, as shownby the angle of the input information (106) at the input interface (112)and its angle as it exits substrate (108). FIG. 5E illustrates the samereflector (110) active, with image information coming from the left ofthe field of view, as shown by the angle of the input information (106)at the input interface (112) and the resultant exit angle at the surfaceof the substrate (108). FIG. 5F illustrates a configuration wherein thebottom reflector (110) is active, with image information (106) coming infrom the far right of the overall field of view. For example, FIGS. 5C,5D, and 5E can illustrate one frame representing the center of aframe-sequential tiled image, and FIG. 5F can illustrate a second framerepresenting the far right of that tiled image.

In one embodiment, the light carrying the image information (106) maystrike the reflective surface (110) directly after entering substrate(108) at input interface (112), without first reflecting from thesurfaces of substrate (108). In one embodiment, the light carrying theimage information (106) may reflect from one or more surfaces ofsubstrate (108) after entering at input interface (112) and beforestriking the reflective surface (110); for instance, substrate (108) mayact as a planar waveguide, propagating the light carrying imageinformation (106) by total internal reflection. Light may also reflectfrom one or more surfaces of the substrate (108) from a partiallyreflective coating, a wavelength-selective coating, an angle-selectivecoating, and/or a polarization-selective coating.

In one embodiment, the angled reflectors may be constructed using anelectro-active material, such that upon application of a voltage and/orcurrent to a particular reflector, the refractive index of the materialcomprising such reflector changes from an index substantially matched tothe rest of the substrate (108), in which case the reflector is in atransmissive configuration, to a reflective configuration wherein therefractive index of the reflector mismatches the refractive index of thesubstrate (108) such that a reflection effect is created. Exampleelectro-active material includes lithium niobate and electro-activepolymers. Suitable substantially transparent electrodes for controllinga plurality of such reflectors may comprise materials such as indium tinoxide, which is utilized in liquid crystal displays.

In one embodiment, the electro-active reflectors (110) may compriseliquid crystal, embedded in a substrate (108) host medium such as glassor plastic. In some variations, liquid crystal may be selected thatchanges refractive index as a function of an applied electric signal, sothat more analog changes may be accomplished as opposed to binary (fromone transmissive state to one reflective state). In an embodimentwherein 6 sub-images are to be presented to the eye frame-sequential toform a large tiled image with an overall refresh rate of 60 frames persecond, it is desirable to have an input display that can refresh at therate of about 360 Hz, with an electro-active reflector array that cankeep up with such frequency. In one embodiment, lithium niobate may beutilized as an electro-active reflective material as opposed to liquidcrystal; lithium niobate is utilized in the photonics industry forhigh-speed switches and fiber optic networks and has the capability toswitch refractive index in response to an applied voltage at a very highfrequency; this high frequency may be used to steer line-sequential orpixel-sequential sub-image information, especially if the input displayis a scanned light display, such as a fiber-scanned display or scanningmirror-based display.

In another embodiment, a variable switchable angled mirror configurationmay comprise one or more high-speed mechanically repositionablereflective surfaces, such as a MEMS (micro-electro-mechanical system)device. A MEMS device may include what is known as a “digital mirrordevice”, or “DMD”, (often part of a “digital light processing”, or “DLP”system, such as those available from Texas Instruments, Inc.). Inanother electromechanical embodiment, a plurality of air-gapped (or invacuum) reflective surfaces could be mechanically moved in and out ofplace at high frequency. In another electromechanical embodiment, asingle reflective surface may be moved up and down and re-pitched atvery high frequency.

Referring to FIG. 5G, it is notable that the switchable variable anglereflector configurations described herein are capable of passing notonly collimated or flat wavefront information to the retina (54) of theeye (58), but also curved wavefront (122) image information, as shown inthe illustration of FIG. 5G. This generally is not the case with otherwaveguide-based configurations, wherein total internal reflection ofcurved wavefront information causes undesirable complications, andtherefore the inputs generally must be collimated. The ability to passcurved wavefront information facilitates the ability of configurationssuch as those shown in FIGS. 5B-5H to provide the retina (54) with inputperceived as focused at various distances from the eye (58), not justoptical infinity (which would be the interpretation of collimated lightabsent other cues).

Referring to FIG. 5H, in another embodiment, an array of staticpartially reflective surfaces (116) (i.e., always in a reflective mode;in another embodiment, they may be electro-active, as above) may beembedded in a substrate (114) with a high-frequency gating layer (118)controlling outputs to the eye (58) by only allowing transmissionthrough an aperture (120) which is controllably movable. In other words,everything may be selectively blocked except for transmissions throughthe aperture (120). The gating layer (118) may comprise a liquid crystalarray, a lithium niobate array, an array of MEMS shutter elements, anarray of DLP DMD elements, or an array of other MEMS devices configuredto pass or transmit with relatively high-frequency switching and hightransmissibility upon being switched to transmission mode.

Referring to FIGS. 6A-6B, other embodiments are depicted wherein arrayedoptical elements may be combined with exit pupil expansionconfigurations to assist with the comfort of the virtual or augmentedreality experience of the user. With a larger “exit pupil” for theoptics configuration, the user's eye positioning relative to the display(which, as in FIGS. 4A-4D, may be mounted on the user's head in aneyeglasses sort of configuration) is not as likely to disrupt hisexperience—because due to the larger exit pupil of the system, there isa larger acceptable area wherein the user's anatomical pupil may belocated to still receive the information from the display system asdesired. In other words, with a larger exit pupil, the system is lesslikely to be sensitive to slight misalignments of the display relativeto the user's anatomical pupil, and greater comfort for the user may beachieved through less geometric constraint on his or her relationshipwith the display/glasses.

As shown in FIG. 6A, the display (140) on the left feeds a set ofparallel rays into the substrate (124). In one embodiment, the displaymay be a scanned fiber display scanning a narrow beam of light back andforth at an angle as shown to project an image through the lens or otheroptical element (142), which may be utilized to collect theangularly-scanned light and convert it to a parallel bundle of rays. Therays may be reflected from a series of reflective surfaces (126, 128,130, 132, 134, 136) which may be configured to partially reflect andpartially transmit incoming light so that the light may be shared acrossthe group of reflective surfaces (126, 128, 130, 132, 134, 136)approximately equally. With a small lens (138) placed at each exit pointfrom the waveguide (124), the exiting light rays may be steered througha nodal point and scanned out toward the eye (58) to provide an array ofexit pupils, or the functional equivalent of one large exit pupil thatis usable by the user as he or she gazes toward the display system.

For virtual reality configurations wherein it is desirable to also beable to see through the waveguide to the real world (144), a similar setof lenses (139) may be presented on the opposite side of the waveguide(124) to compensate for the lower set of lenses; thus creating a theequivalent of a zero-magnification telescope. The reflective surfaces(126, 128, 130, 132, 134, 136) each may be aligned at approximately 45degrees as shown, or may be configured to have different alignments,akin to the configurations of FIGS. 5B-5H, for example). The reflectivesurfaces (126, 128, 130, 132, 134, 136) may comprisewavelength-selective reflectors, band pass reflectors, half silveredmirrors, or other reflective configurations. The lenses (138, 139) shownare refractive lenses, but diffractive lens elements may also beutilized.

Referring to FIG. 6B, a somewhat similar configuration is depictedwherein a plurality of curved reflective surfaces (148, 150, 152, 154,156, 158) may be utilized to effectively combine the lens (element 138of FIG. 6A) and reflector (elements 126, 128, 130, 132, 134, 136 of FIG.6A) functionality of the embodiment of FIG. 6A, thereby obviating theneed for the two groups of lenses (element 138 of FIG. 6A). The curvedreflective surfaces (148, 150, 152, 154, 156, 158) may be various curvedconfigurations selected to both reflect and impart angular change, suchas parabolic or elliptical curved surfaces. With a parabolic shape, aparallel set of incoming rays will be collected into a single outputpoint; with an elliptical configuration, a set of rays diverging from asingle point of origin are collected to a single output point. As withthe configuration of FIG. 6A, the curved reflective surfaces (148, 150,152, 154, 156, 158) preferably are configured to partially reflect andpartially transmit so that the incoming light is shared across thelength of the waveguide (146). The curved reflective surfaces (148, 150,152, 154, 156, 158) may comprise wavelength-selective notch reflectors,half silvered mirrors, or other reflective configurations. In anotherembodiment, the curved reflective surfaces (148, 150, 152, 154, 156,158) may be replaced with diffractive reflectors configured to reflectand also deflect.

Referring to FIG. 7A, perceptions of Z-axis difference (i.e., distancestraight out from the eye along the optical axis) may be facilitated byusing a waveguide in conjunction with a variable focus optical elementconfiguration. As shown in FIG. 7A, image information from a display(160) may be collimated and injected into a waveguide (164) anddistributed in a large exit pupil manner using, e.g., configurationssuch as those described in reference to FIGS. 6A and 6B, or othersubstrate-guided optics methods known to those skilled in the art—andthen variable focus optical element capability may be utilized to changethe focus of the wavefront of light emerging from the waveguide andprovide the eye with the perception that the light coming from thewaveguide (164) is from a particular focal distance. In other words,since the incoming light has been collimated to avoid challenges intotal internal reflection waveguide configurations, it will exit incollimated fashion, requiring a viewer's eye to accommodate to the farpoint to bring it into focus on the retina, and naturally be interpretedas being from optical infinity—unless some other intervention causes thelight to be refocused and perceived as from a different viewingdistance; one suitable such intervention is a variable focus lens.

In the embodiment of FIG. 7A, collimated image information is injectedinto a piece of glass (162) or other material at an angle such that ittotally internally reflects and is passed into the adjacent waveguide(164). The waveguide (164) may be configured akin to the waveguides ofFIG. 6A or 6B (124, 146, respectively) so that the collimated light fromthe display is distributed to exit somewhat uniformly across thedistribution of reflectors or diffractive features along the length ofthe waveguide. Upon exit toward the eye (58), in the depictedconfiguration the exiting light is passed through a variable focus lenselement (166) wherein, depending upon the controlled focus of thevariable focus lens element (166), the light exiting the variable focuslens element (166) and entering the eye (58) will have various levels offocus (a collimated flat wavefront to represent optical infinity, moreand more beam divergence/wavefront curvature to represent closer viewingdistance relative to the eye 58).

To compensate for the variable focus lens element (166) between the eye(58) and the waveguide (164), another similar variable focus lenselement (167) is placed on the opposite side of the waveguide (164) tocancel out the optical effects of the lenses (166) for light coming fromthe world (144) for augmented reality (i.e., as described above, onelens compensates for the other, producing the functional equivalent of azero-magnification telescope).

The variable focus lens element (166) may be a refractive element, suchas a liquid crystal lens, an electro-active lens, a conventionalrefractive lens with moving elements, a mechanical-deformation-basedlens (such as a fluid-filled membrane lens, or a lens akin to the humancrystalline lens wherein a flexible element is flexed and relaxed byactuators), an electrowetting lens, or a plurality of fluids withdifferent refractive indices. The variable focus lens element (166) mayalso comprise a switchable diffractive optical element (such as onefeaturing a polymer dispersed liquid crystal approach wherein a hostmedium, such as a polymeric material, has microdroplets of liquidcrystal dispersed within the material; when a voltage is applied, themolecules reorient so that their refractive indices no longer match thatof the host medium, thereby creating a high-frequency switchablediffraction pattern).

One embodiment includes a host medium in which microdroplets of a Kerreffect-based electro-active material, such as lithium niobate, isdispersed within the host medium, enabling refocusing of imageinformation on a pixel-by-pixel or line-by-line basis, when coupled witha scanning light display, such as a fiber-scanned display orscanning-mirror-based display. In a variable focus lens element (166)configuration wherein liquid crystal, lithium niobate, or othertechnology is utilized to present a pattern, the pattern spacing may bemodulated to not only change the focal power of the variable focus lenselement (166), but also to change the focal power of the overall opticalsystem—for a zoom lens type of functionality.

In one embodiment, the lenses (166) could be telecentric, in that focusof the display imagery can be altered while keeping magnificationconstant—in the same way that a photography zoom lens may be configuredto decouple focus from zoom position. In another embodiment, the lenses(166) may be non-telecentric, so that focus changes will also slave zoomchanges. With such a configuration, such magnification changes may becompensated for in software with dynamic scaling of the output from thegraphics system in sync with focus changes).

Referring back to the projector or other video display unit (160) andthe issue of how to feed images into the optical display system, in a“frame sequential” configuration, a stack of sequential two-dimensionalimages may be fed to the display sequentially to producethree-dimensional perception over time; in a manner akin to the mannerin which a computed tomography system uses stacked image slices torepresent a three-dimensional structure. A series of two-dimensionalimage slices may be presented to the eye, each at a different focaldistance to the eye, and the eye/brain would integrate such a stack intoa perception of a coherent three-dimensional volume. Depending upon thedisplay type, line-by-line, or even pixel-by-pixel sequencing may beconducted to produce the perception of three-dimensional viewing. Forexample, with a scanned light display (such as a scanning fiber displayor scanning mirror display), then the display is presenting thewaveguide (164) with one line or one pixel at a time in a sequentialfashion.

If the variable focus lens element (166) is able to keep up with thehigh-frequency of pixel-by-pixel or line-by-line presentation, then eachline or pixel may be presented and dynamically focused through thevariable focus lens element (166) to be perceived at a different focaldistance from the eye (58). Pixel-by-pixel focus modulation generallyrequires an extremely fast/high-frequency variable focus lens element(166). For example, a 1080P resolution display with an overall framerate of 60 frames per second typically presents around 125 millionpixels per second. Such a configuration also may be constructed using asolid state switchable lens, such as one using an electro-activematerial, e.g., lithium niobate or an electro-active polymer. Inaddition to its compatibility with the system illustrated in FIG. 7A, aframe sequential multi-focal display driving approach may be used inconjunction with a number of the display system and optics embodimentsdescribed in this disclosure.

Referring to FIG. 7B, with an electro-active layer (172) (such as onecomprising liquid crystal or lithium niobate) surrounded by functionalelectrodes (170, 174) which may be made of indium tin oxide, a waveguide(168) with a conventional transmissive substrate (176, such as one madefrom glass or plastic with known total internal reflectioncharacteristics and an index of refraction that matches the on or offstate of the electro-active layer 172) may be controlled such that thepaths of entering beams may be dynamically altered to essentially createa time-varying light field.

Referring to FIG. 8A, a stacked waveguide assembly (178) may be utilizedto provide three-dimensional perception to the eye/brain by having aplurality of waveguides (182, 184, 186, 188, 190) and a plurality ofweak lenses (198, 196, 194, 192) configured together to send imageinformation to the eye with various levels of wavefront curvature foreach waveguide level indicative of focal distance to be perceived forthat waveguide level. A plurality of displays (200, 202, 204, 206, 208),or in another embodiment a single multiplexed display, may be utilizedto inject collimated image information into the waveguides (182, 184,186, 188, 190), each of which may be configured, as described above, todistribute incoming light substantially equally across the length ofeach waveguide, for exit down toward the eye.

The waveguide (182) nearest the eye is configured to deliver collimatedlight, as injected into such waveguide (182), to the eye, which may berepresentative of the optical infinity focal plane. The next waveguideup (184) is configured to send out collimated light which passes throughthe first weak lens (192; e.g., a weak negative lens) before it canreach the eye (58); such first weak lens (192) may be configured tocreate a slight convex wavefront curvature so that the eye/braininterprets light coming from that next waveguide up (184) as coming froma first focal plane closer inward toward the person from opticalinfinity. Similarly, the third up waveguide (186) passes its outputlight through both the first (192) and second (194) lenses beforereaching the eye (58); the combined optical power of the first (192) andsecond (194) lenses may be configured to create another incrementalamount of wavefront divergence so that the eye/brain interprets lightcoming from that third waveguide up (186) as coming from a second focalplane even closer inward toward the person from optical infinity thanwas light from the next waveguide up (184).

The other waveguide layers (188, 190) and weak lenses (196, 198) aresimilarly configured, with the highest waveguide (190) in the stacksending its output through all of the weak lenses between it and the eyefor an aggregate focal power representative of the closest focal planeto the person. To compensate for the stack of lenses (198, 196, 194,192) when viewing/interpreting light coming from the world (144) on theother side of the stacked waveguide assembly (178), a compensating lenslayer (180) is disposed at the top of the stack to compensate for theaggregate power of the lens stack (198, 196, 194, 192) below. Such aconfiguration provides as many perceived focal planes as there areavailable waveguide/lens pairings, again with a relatively large exitpupil configuration as described above. Both the reflective aspects ofthe waveguides and the focusing aspects of the lenses may be static(i.e., not dynamic or electro-active). In an alternative embodiment theymay be dynamic using electro-active features as described above,enabling a small number of waveguides to be multiplexed in a timesequential fashion to produce a larger number of effective focal planes.

Referring to FIGS. 8B-8N, various aspects of diffraction configurationsfor focusing and/or redirecting collimated beams are depicted. Otheraspects of diffraction systems for such purposes are disclosed in U.S.Patent Application Ser. No. 61/845,907 (U.S. patent application Ser. No.14/331,218), which is incorporated by reference herein in its entirety.Referring to FIG. 8B, passing a collimated beam through a lineardiffraction pattern (210), such as a Bragg grating, will deflect, or“steer”, the beam. Passing a collimated beam through a radiallysymmetric diffraction pattern (212), or “Fresnel zone plate”, willchange the focal point of the beam. FIG. 8C illustrates the deflectioneffect of passing a collimated beam through a linear diffraction pattern(210); FIG. 8D illustrates the focusing effect of passing a collimatedbeam through a radially symmetric diffraction pattern (212).

Referring to FIGS. 8E and 8F, a combination diffraction pattern that hasboth linear and radial elements (214) produces both deflection andfocusing of a collimated input beam. These deflection and focusingeffects can be produced in a reflective as well as transmissive mode.These principles may be applied with waveguide configurations to allowfor additional optical system control, as shown in FIGS. 8G-8N, forexample. As shown in FIGS. 8G-8N, a diffraction pattern (220), or“diffractive optical element” (or “DOE”) has been embedded within aplanar waveguide (216) such that as a collimated beam is totallyinternally reflected along the planar waveguide (216), it intersects thediffraction pattern (220) at a multiplicity of locations.

Preferably, the DOE (220) has a relatively low diffraction efficiency sothat only a portion of the light of the beam is deflected away towardthe eye (58) with each intersection of the DOE (220) while the restcontinues to move through the planar waveguide (216) via total internalreflection; the light carrying the image information is thus dividedinto a number of related light beams that exit the waveguide at amultiplicity of locations and the result is a fairly uniform pattern ofexit emission toward the eye (58) for this particular collimated beambouncing around within the planar waveguide (216), as shown in FIG. 8H.The exit beams toward the eye (58) are shown in FIG. 8H as substantiallyparallel, because, in this case, the DOE (220) has only a lineardiffraction pattern. As shown in the comparison between FIGS. 8L, 8M,and 8N, changes to this linear diffraction pattern pitch may be utilizedto controllably deflect the exiting parallel beams, thereby producing ascanning or tiling functionality.

Referring back to FIG. 8I, with changes in the radially symmetricdiffraction pattern component of the embedded DOE (220), the exit beampattern is more divergent, which would require the eye to accommodationto a closer distance to bring it into focus on the retina and would beinterpreted by the brain as light from a viewing distance closer to theeye than optical infinity. Referring to FIG. 8J, with the addition ofanother waveguide (218) into which the beam may be injected (by aprojector or display, for example), a DOE (221) embedded in this otherwaveguide (218), such as a linear diffraction pattern, may function tospread the light across the entire larger planar waveguide (216), whichfunctions to provide the eye (58) with a very large incoming field ofincoming light that exits from the larger planar waveguide (216), i.e.,a large eye box, in accordance with the particular DOE configurations atwork.

The DOEs (220, 221) are depicted bisecting the associated waveguides(216, 218) but this need not be the case; they could be placed closerto, or upon, either side of either of the waveguides (216, 218) to havethe same functionality. Thus, as shown in FIG. 8K, with the injection ofa single collimated beam, an entire field of cloned collimated beams maybe directed toward the eye (58). In addition, with a combined lineardiffraction pattern/radially symmetric diffraction pattern scenario suchas that depicted in FIGS. 8F (214) and 8I (220), a beam distributionwaveguide optic (for functionality such as exit pupil functionalexpansion; with a configuration such as that of FIG. 8K, the exit pupilcan be as large as the optical element itself, which can be a verysignificant advantage for user comfort and ergonomics) with Z-axisfocusing capability is presented, in which both the divergence angle ofthe cloned beams and the wavefront curvature of each beam representlight coming from a point closer than optical infinity.

In one embodiment, one or more DOEs are switchable between “on” statesin which they actively diffract, and “off” states in which they do notsignificantly diffract. For instance, a switchable DOE may comprise alayer of polymer dispersed liquid crystal, in which microdropletscomprise a diffraction pattern in a host medium, and the refractiveindex of the microdroplets can be switched to substantially match therefractive index of the host material (in which case the pattern doesnot appreciably diffract incident light) or the microdroplet can beswitched to an index that does not match that of the host medium (inwhich case the pattern actively diffracts incident light). Further, withdynamic changes to the diffraction terms, such as the linear diffractionpitch term as in FIGS. 8L-8N, a beam scanning or tiling functionalitymay be achieved. As noted above, it is desirable to have a relativelylow diffraction grating efficiency in each of the DOEs (220, 221)because it facilitates distribution of the light, and also because lightcoming through the waveguides that is desirably transmitted (forexample, light coming from the world 144 toward the eye 58 in anaugmented reality configuration) is less affected when the diffractionefficiency of the DOE that it crosses (220) is lower—so a better view ofthe real world through such a configuration is achieved.

Configurations such as those illustrated in FIG. 8K preferably aredriven with injection of image information in a time sequentialapproach, with frame sequential driving being the most straightforwardto implement. For example, an image of the sky at optical infinity maybe injected at time1 and the diffraction grating retaining collimationof light may be utilized; then an image of a closer tree branch may beinjected at time2 while a DOE controllably imparts a focal change, sayone diopter or 1 meter away, to provide the eye/brain with theperception that the branch light information is coming from the closerfocal range. This kind of paradigm can be repeated in rapid timesequential fashion such that the eye/brain perceives the input to be allpart of the same image. This is just a two focal plane example;preferably the system will be configured to have more focal planes toprovide a smoother transition between objects and their focal distances.This kind of configuration generally assumes that the DOE is switched ata relatively low speed (i.e., in sync with the frame-rate of the displaythat is injecting the images—in the range of tens to hundreds ofcycles/second).

The opposite extreme may be a configuration wherein DOE elements canshift focus at tens to hundreds of MHz or greater, which facilitatesswitching of the focus state of the DOE elements on a pixel-by-pixelbasis as the pixels are scanned into the eye (58) using a scanned lightdisplay type of approach. This is desirable because it means that theoverall display frame-rate can be kept quite low; just low enough tomake sure that “flicker” is not a problem (in the range of about 60-120frames/sec).

In between these ranges, if the DOEs can be switched at KHz rates, thenon a line-by-line basis the focus on each scan line may be adjusted,which may afford the user with a visible benefit in terms of temporalartifacts during an eye motion relative to the display, for example. Forinstance, the different focal planes in a scene may, in this manner, beinterleaved, to minimize visible artifacts in response to a head motion(as is discussed in greater detail later in this disclosure). Aline-by-line focus modulator may be operatively coupled to a line scandisplay, such as a grating light valve display, in which a linear arrayof pixels is swept to form an image; and may be operatively coupled toscanned light displays, such as fiber-scanned displays andmirror-scanned light displays.

A stacked configuration, similar to those of FIG. 8A, may use dynamicDOEs (rather than the static waveguides and lenses of the embodiment ofFIG. 8A) to provide multi-planar focusing simultaneously. For example,with three simultaneous focal planes, a primary focus plane (based uponmeasured eye accommodation, for example) could be presented to the user,and a +margin and −margin (i.e., one focal plane closer, one fartherout) could be utilized to provide a large focal range in which the usercan accommodate before the planes need be updated. This increased focalrange can provide a temporal advantage if the user switches to a closeror farther focus (i.e., as determined by accommodation measurement);then the new plane of focus could be made to be the middle depth offocus, with the + and − margins again ready for a fast switchover toeither one while the system catches up.

Referring to FIG. 8O, a stack (222) of planar waveguides (244, 246, 248,250, 252) is shown, each having a reflector (254, 256, 258, 260, 262) atthe end and being configured such that collimated image informationinjected in one end by a display (224, 226, 228, 230, 232) bounces bytotal internal reflection down to the reflector, at which point some orall of the light is reflected out toward an eye or other target. Each ofthe reflectors may have slightly different angles so that they allreflect exiting light toward a common destination such as a pupil. Sucha configuration is somewhat similar to that of FIG. 5B, with theexception that each different angled reflector in the embodiment of FIG.8O has its own waveguide for less interference when projected light istravelling to the targeted reflector. Lenses (234, 236, 238, 240, 242)may be interposed between the displays and waveguides for beam steeringand/or focusing.

FIG. 8P illustrates a geometrically staggered version wherein reflectors(276, 278, 280, 282, 284) are positioned at staggered lengths in thewaveguides (266, 268, 270, 272, 274) so that exiting beams may berelatively easily aligned with objects such as an anatomical pupil. Withknowledge of how far the stack (264) is going to be from the eye (suchas 28 mm between the cornea of the eye and an eyeglasses lens, a typicalcomfortable geometry), the geometries of the reflectors (276, 278, 280,282, 284) and waveguides (266, 268, 270, 272, 274) may be set up to fillthe eye pupil (typically about 8 mm across or less) with exiting light.By directing light to an eye box larger than the diameter of the eyepupil, the viewer may make eye movements while retaining the ability tosee the displayed imagery. Referring back to the discussion related toFIGS. 5A and 5B about field of view expansion and reflector size, anexpanded field of view is presented by the configuration of FIG. 8P aswell, and it does not involve the complexity of the switchablereflective elements of the embodiment of FIG. 5B.

FIG. 8Q illustrates a version wherein many reflectors (298) form arelatively continuous curved reflection surface in the aggregate ordiscrete flat facets that are oriented to align with an overall curve.The curve could a parabolic or elliptical curve and is shown cuttingacross a plurality of waveguides (288, 290, 292, 294, 296) to minimizeany crosstalk issues, although it also could be utilized with amonolithic waveguide configuration.

In one implementation, a high-frame-rate and lower persistence displaymay be combined with a lower-frame-rate and higher persistence displayand a variable focus element to comprise a relatively high-frequencyframe sequential volumetric display. In one embodiment, thehigh-frame-rate display has a lower bit depth and the lower-frame-ratedisplay has a higher bit depth, and are combined to comprise aneffective high-frame-rate and high bit depth display, that is wellsuited to presenting image slices in a frame sequential fashion. Withsuch an approach, a three-dimensional volume that is desirablyrepresented is functionally divided into a series of two-dimensionalslices. Each of those two-dimensional slices is projected to the eyeframe sequentially, and in sync with this presentation, the focus of avariable focus element is changed.

In one embodiment, to get enough frame rate to support such aconfiguration, two display elements may be integrated: a full-color,high-resolution liquid crystal display (“LCD”; a backlightedferroelectric panel display also may be utilized in another embodiment;in a further embodiment a scanning fiber display may be utilized)operating at 60 frames per second, and aspects of a higher-frequency DLPsystem. Instead of illuminating the back of the LCD panel in aconventional manner (i.e., with a full size fluorescent lamp or LEDarray), the conventional lighting configuration may be removed toaccommodate using the DLP projector to project a mask pattern on theback of the LCD (in one embodiment, the mask pattern may be binary inthat the DLP either projects illumination, or not-illumination; inanother embodiment described below, the DLP may be utilized to project agrayscale mask image).

DLP projection systems can operate at very high frame rates; in oneembodiment for 6 depth planes at 60 frames per second, a DLP projectionsystem can be operated against the back of the LCD display at 360frames/second. Then the DLP projector is utilized to selectivelyilluminate portions of the LCD panel in sync with a high-frequencyvariable focus element (such as a deformable membrane mirror) that isdisposed between the viewing side of the LCD panel and the eye of theuser, the variable focus element being used to change the global displayfocus on a frame by frame basis at 360 frames/second. In one embodiment,the variable focus element is positioned to be optically conjugate tothe exit pupil, to enable adjustments of focus without simultaneouslyaffecting image magnification or “zoom.” In another embodiment, thevariable focus element is not conjugate to the exit pupil, such thatimage magnification changes accompany focus adjustments, and software isused to compensate for these optical magnification changes and anydistortions by pre-scaling or warping the images to be presented.

Operationally, it's useful to consider an example again wherein athree-dimensional scene is to be presented to a user wherein the sky inthe background is to be at a viewing distance of optical infinity, andwherein a branch coupled to a tree located at a certain location closerto the user than optical infinity extends from the tree trunk in adirection toward the user, so that the tip of the branch is closer tothe user than is the proximal portion of the branch that joins the treetrunk.

In one embodiment, for a given global frame, the system may beconfigured to present on an LCD a full-color, all in-focus image of thetree branch in front the sky. Then at subframe1, within the globalframe, the DLP projector in a binary masking configuration (i.e.,illumination or absence of illumination) may be used to only illuminatethe portion of the LCD that represents the cloudy sky while functionallyblack-masking (i.e., failing to illuminate) the portion of the LCD thatrepresents the tree branch and other elements that are not to beperceived at the same focal distance as the sky, and the variable focuselement (such as a deformable membrane mirror) may be utilized toposition the focal plane at optical infinity so that the eye sees asub-image at subframe1 as being clouds that are infinitely far away.

Then at subframe2, the variable focus element may be switched tofocusing on a point about 1 meter away from the user's eyes (or whateverdistance is required; here 1 meter for the branch location is used forillustrative purposes), the pattern of illumination from the DLP can beswitched so that the system only illuminates the portion of the LCD thatrepresents the tree branch while functionally black-masking (i.e.,failing to illuminate) the portion of the LCD that represents the skyand other elements that are not to be perceived at the same focaldistance as the tree branch. Thus the eye gets a quick flash of cloud atoptical infinity followed by a quick flash of tree at 1 meter, and thesequence is integrated by the eye/brain to form a three-dimensionalperception. The branch may be positioned diagonally relative to theviewer, such that it extends through a range of viewing distances, e.g.,it may join with the trunk at around 2 meters viewing distance while thetips of the branch are at the closer position of 1 meter.

In this case, the display system can divide the 3-D volume of the treebranch into multiple slices, rather than a single slice at 1 meter. Forinstance, one focus slice may be used to represent the sky (using theDLP to mask all areas of the tree during presentation of this slice),while the tree branch is divided across 5 focus slices (using the DLP tomask the sky and all portions of the tree except one, for each part ofthe tree branch to be presented). Preferably, the depth slices arepositioned with a spacing equal to or smaller than the depth of focus ofthe eye, such that the viewer will be unlikely to notice the transitionbetween slices, and instead perceive a smooth and continuous flow of thebranch through the focus range.

In another embodiment, rather than utilizing the DLP in a binary(illumination or darkfield only) mode, it may be utilized to project agrayscale (for example, 256 shades of grayscale) mask onto the back ofthe LCD panel to enhance three-dimensional perception. The grayscaleshades may be utilized to impart to the eye/brain a perception thatsomething resides in between adjacent depth or focal planes. Back to thebranch and clouds scenario, if the leading edge of the branch closest tothe user is to be in focalplane1, then at subframe1, that portion branchon the LCD may be lit up with full intensity white from the DLP systemwith the variable focus element at focalplane1.

Then at subframe2, with the variable focus element at focalplane2 rightbehind the part that was lit up, there would be no illumination. Theseare similar steps to the binary DLP masking configuration above.However, if there is a portion of the branch that is to be perceived ata position between focalplane1 and focalplane1, e.g., halfway, grayscalemasking can be utilized. The DLP can project an illumination mask tothat portion during both subframe1 and subframe2, but athalf-illumination (such as at level 128 out of 256 grayscale) for eachsubframe. This provides the perception of a blending of depth of focuslayers, with the perceived focal distance being proportional to theilluminance ratio between subframe1 and subframe2. For instance, for aportion of the tree branch that should lie ¾ths of the way betweenfocalplane1 and focalplane2, an about 25% intensity grayscale mask canbe used to illuminate that portion of the LCD at subframe1 and an about75% grayscale mask can be used to illuminate the same portion of the LCDat subframe2.

In one embodiment, the bit depths of both the low-frame-rate display andthe high-frame-rate display can be combined for image modulation, tocreate a high dynamic range display. The high dynamic range driving maybe conducted in tandem with the focus plane addressing functiondescribed above, to comprise a high dynamic range multi-focal 3-Ddisplay.

In another embodiment that may be more efficient on computationresources, only a certain portion of the display (i.e., LCD) output maybe mask-illuminated by the DMD and variably focused en route to theuser's eye. For example, the middle portion of the display may be maskilluminated, with the periphery of the display not providing varyingaccommodation cues to the user (i.e. the periphery could be uniformlyilluminated by the DLP DMD, while a central portion is actively maskedand variably focused en route to the eye).

In the above described embodiment, a refresh rate of about 360 Hz allowsfor 6 depth planes at about 60 frames/second each. In anotherembodiment, even higher refresh rates may be achieved by increasing theoperating frequency of the DLP. A standard DLP configuration uses a MEMSdevice and an array of micro-mirrors that toggle between a mode ofreflecting light toward the display or user to a mode of reflectinglight away from the display or user, such as into a light trap—thus theyare inherently binary. DLPs typically create grayscale images using apulse width modulation schema wherein the mirror is left in the “on”state for a variable amount of time for a variable duty cycle in orderto create a brighter pixel, or pixel of interim brightness. Thus, tocreate grayscale images at moderate frame rate, they are running at amuch higher binary rate.

In the above described configurations, such setup works well forcreating grayscale masking. However, if the DLP drive scheme is adaptedso that it is flashing subimages in a binary pattern, then the framerate may be increased significantly—by thousands of frames per second,which allows for hundreds to thousands of depth planes being refreshedat 60 frames/second, which may be utilized to obviate thebetween-depth-plane grayscale interpolating as described above. Atypical pulse width modulation scheme for a Texas Instruments DLP systemhas an 8-bit command signal (first bit is the first long pulse of themirror; second bit is a pulse that is half as long as the first; thirdbit is half as long again; and so on)—so that the configuration cancreate 2 to the 8th power different illumination levels. In oneembodiment, the backlighting from the DLP may have its intensity variedin sync with the different pulses of the DMD to equalize the brightnessof the subimages that are created, which is a practical workaround toget existing DMD drive electronics to produce significantly higher framerates.

In another embodiment, direct control changes to the DMD driveelectronics and software may be utilized to have the mirrors always havean equal on-time instead of the variable on-time configuration that isconventional, which would facilitate higher frame rates. In anotherembodiment, the DMD drive electronics may be configured to present lowbit depth images at a frame rate above that of high bit depth images butlower than the binary frame rate, enabling some grayscale blendingbetween focus planes, while moderately increasing the number of focusplanes.

In another embodiment, when limited to a finite number of depth planes,such as 6 in the example above, it is desirable to functionally movethese 6 depth planes around to be maximally useful in the scene that isbeing presented to the user. For example, if a user is standing in aroom and a virtual monster is to be placed into his augmented realityview, the virtual monster being about 2 feet deep in the Z axis straightaway from the user's eyes, then it makes sense to cluster all 6 depthplanes around the center of the monster's current location (anddynamically move them with him as he moves relative to the user)—so thatmore rich accommodation cues may be provided for the user, with all sixdepth planes in the direct region of the monster (for example, 3 infront of the center of the monster, 3 in back of the center of themonster). Such allocation of depth planes is content dependent.

For example, in the scene above the same monster is to be presented inthe same room, but also to be presented to the user is a virtual windowframe element, and then a virtual view to optical infinity out of thevirtual window frame, it will be useful to spend at least one depthplane on optical infinity, one on the depth of the wall that is to housethe virtual window frame, and then perhaps the remaining four depthplanes on the monster in the room. If the content causes the virtualwindow to disappear, then the two depth planes may be dynamicallyreallocated to the region around the monster, and so on—content-baseddynamic allocation of focal plane resources to provide the most richexperience to the user given the computing and presentation resources.

In another embodiment, phase delays in a multicore fiber or an array ofsingle-core fibers may be utilized to create variable focus lightwavefronts. Referring to FIG. 9A, a multicore fiber (300) may comprisethe aggregation of multiple individual fibers (302); FIG. 9B shows aclose-up view of a multicore assembly, which emits light from each corein the form of a spherical wavefront (304) from each. If the cores aretransmitting coherent light, e.g., from a shared laser light source,these small spherical wavefronts ultimately constructively anddestructively interfere with each other, and if they were emitted fromthe multicore fiber in phase, they will develop an approximately planarwavefront (306) in the aggregate, as shown. However, if phase delays areinduced between the cores (using a conventional phase modulator such asone using lithium niobate, for example, to slow the path of some coresrelative to others), then a curved or spherical wavefront may be createdin the aggregate, to represent at the eyes/brain an object coming from apoint closer than optical infinity, which presents another option thatmay be used in place of the variable focus elements described above. Inother words, such a phased multicore configuration, or phased array, maybe utilized to create multiple optical focus levels from a light source.

In another embodiment related to the use of optical fibers, a knownFourier transform aspect of multi-mode optical fiber or light guidingrods or pipes may be utilized for control of the wavefronts that areoutput from such fiber. Optical fibers typically are available in twocategories: single mode and multi-mode. Multi-mode optical fibertypically has larger core diameters and allows light to propagate alongmultiple angular paths, rather than just the one of single mode opticalfiber. It is known that if an image is injected into one end of amulti-mode fiber, that angular differences that are encoded into thatimage will be retained to some degree as it propagates through themulti-mode fiber, and for some configurations the output from the fiberwill be significantly similar to a Fourier transform of the image thatwas input.

Thus in one embodiment, the inverse Fourier transform of a wavefront(such as a diverging spherical wavefront to represent a focal planenearer to the user than optical infinity) may be input so that, afterpassing through the fiber that optically imparts a Fourier transform,the output is the desired shaped, or focused, wavefront. Such output endmay be scanned about to be used as a scanned fiber display, or may beused as a light source for a scanning mirror to form an image, forinstance. Thus such a configuration may be utilized as yet another focusmodulation subsystem. Other kinds of light patterns and wavefronts maybe injected into a multi-mode fiber, such that on the output end, acertain spatial pattern is emitted. This may be utilized to have theequivalent of a wavelet pattern (in optics, an optical system may beanalyzed in terms of what are called the Zernicke coefficients; imagesmay be similarly characterized and decomposed into smaller principalcomponents, or a weighted combination of comparatively simpler imagecomponents). Thus if light is scanned into the eye using the principalcomponents on the input side, a higher resolution image may be recoveredat the output end of the multi-mode fiber.

In another embodiment, the Fourier transform of a hologram may beinjected into the input end of a multi-mode fiber to output a wavefrontthat may be used for three-dimensional focus modulation and/orresolution enhancement. Certain single fiber core, multi-core fibers, orconcentric core+cladding configurations also may be utilized in theaforementioned inverse Fourier transform configurations.

In another embodiment, rather than physically manipulating thewavefronts approaching the eye of the user at a high frame rate withoutregard to the user's particular state of accommodation or eye gaze, asystem may be configured to monitor the user's accommodation and ratherthan presenting a set of multiple different light wavefronts, present asingle wavefront at a time that corresponds to the accommodation stateof the eye. Accommodation may be measured directly (such as by infraredautorefractor or eccentric photorefraction) or indirectly (such as bymeasuring the convergence level of the two eyes of the user; asdescribed above, vergence and accommodation are strongly linkedneurologically, so an estimate of accommodation can be made based uponvergence geometry). Thus with a determined accommodation of, say, 1meter from the user, then the wavefront presentations at the eye may beconfigured for a 1 meter focal distance using any of the above variablefocus configurations. If an accommodation change to focus at 2 meters isdetected, the wavefront presentation at the eye may be reconfigured fora 2 meter focal distance, and so on.

Thus in one embodiment incorporating accommodation tracking, a variablefocus element may be placed in the optical path between an outputtingcombiner (e.g., a waveguide or beamsplitter) and the eye of the user, sothat the focus may be changed along with (i.e., preferably at the samerate as) accommodation changes of the eye. Software effects may beutilized to produce variable amounts blur (e.g., Gaussian) to objectswhich should not be in focus to simulate the dioptric blur expected atthe retina if an object were at that viewing distance and enhance thethree-dimensional perception by the eyes/brain.

A simple embodiment is a single plane whose focus level is slaved to theviewer's accommodation level, however the performance demands on theaccommodation tracking system can be relaxed if even a low number ofmultiple planes are used. Referring to FIG. 10, in another embodiment, astack (328) of about 3 waveguides (318, 320, 322) may be utilized tocreate three focal planes worth of wavefronts simultaneously. In oneembodiment, the weak lenses (324, 326) may have static focal distances,and a variable focal lens (316) may be slaved to the accommodationtracking of the eyes such that one of the three waveguides (say themiddle waveguide 320) outputs what is deemed to be the in-focuswavefront, while the other two waveguides (322, 318) output a +marginwavefront and a −margin wavefront (i.e., a little farther than detectedfocal distance, a little closer than detected focal distance) which mayimprove the three-dimensional perception and also provide enoughdifference for the brain/eye accommodation control system to sense someblur as negative feedback, which enhances the perception of reality, andallows a range of accommodation before an physical adjustment of thefocus levels is necessary.

A variable focus compensating lens (314) is also shown to ensure thatlight coming in from the real world (144) in an augmented realityconfiguration is not refocused or magnified by the assembly of the stack(328) and output lens (316). The variable focus in the lenses (316, 314)may be achieved, as discussed above, with refractive, diffractive, orreflective techniques.

In another embodiment, each of the waveguides in a stack may containtheir own capability for changing focus (such as by having an includedelectronically switchable DOE) so that the variable focus element neednot be centralized as in the stack (328) of the configuration of FIG.10.

In another embodiment, variable focus elements may be interleavedbetween the waveguides of a stack (i.e., rather than fixed focus weaklenses as in the embodiment of FIG. 10) to obviate the need for acombination of fixed focus weak lenses plus whole-stack-refocusingvariable focus element.

Such stacking configurations may be used in accommodation trackedvariations as described herein, and also in a frame-sequentialmulti-focal display approach.

In a configuration wherein light enters the pupil with a small exitpupil, such as ½ mm diameter or less, one has the equivalent of apinhole lens configuration wherein the beam is always interpreted asin-focus by the eyes/brain—e.g., a scanned light display using a 0.5 mmdiameter beam to scan images to the eye. Such a configuration is knownas a Maxwellian view configuration, and in one embodiment, accommodationtracking input may be utilized to induce blur using software to imageinformation that is to be perceived as at a focal plane behind or infront of the focal plane determined from the accommodation tracking. Inother words, if one starts with a display presenting a Maxwellian view,then everything theoretically can be in focus, and to provide a rich andnatural three-dimensional perception, simulated dioptric blur may beinduced with software, and may be slaved to the accommodation trackingstatus.

In one embodiment a scanning fiber display is well suited to suchconfiguration because it may be configured to only output small-diameterbeams in a Maxwellian form. In another embodiment, an array of smallexit pupils may be created to increase the functional eye box of thesystem (and also to reduce the impact of a light-blocking particle whichmay reside in the vitreous or cornea of the eye), such as by one or morescanning fiber displays, or by a DOE configuration such as thatdescribed in reference to FIG. 8K, with a pitch in the array ofpresented exit pupils that ensure that only one will hit the anatomicalpupil of the user at any given time (for example, if the averageanatomical pupil diameter is 4 mm, one configuration may comprise ½ mmexit pupils spaced at intervals of approximate 4 mm apart). Such exitpupils may also be switchable in response to eye position, such thatonly the eye always receives one, and only one, active small exit pupilat a time; allowing a denser array of exit pupils. Such user will have alarge depth of focus to which software-based blur techniques may beadded to enhance perceived depth perception.

As discussed above, an object at optical infinity creates asubstantially planar wavefront; an object closer, such as 1 m away fromthe eye, creates a curved wavefront (with about 1 m convex radius ofcurvature). The eye's optical system needs to have enough optical powerto bend the incoming rays of light so that they end up focused on theretina (convex wavefront gets turned into concave, and then down to afocal point on the retina). These are basic functions of the eye.

In many of the embodiments described above, light directed to the eyehas been treated as being part of one continuous wavefront, some subsetof which would hit the pupil of the particular eye. In another approach,light directed to the eye may be effectively discretized or broken downinto a plurality of beamlets or individual rays, each of which has adiameter less than about 0.5 mm and a unique propagation pathway as partof a greater aggregated wavefront that may be functionally created withthe an aggregation of the beamlets or rays. For example, a curvedwavefront may be approximated by aggregating a plurality of discreteneighboring collimated beams, each of which is approaching the eye froman appropriate angle to represent a point of origin that matches thecenter of the radius of curvature of the desired aggregate wavefront.

When the beamlets have a diameter of about 0.5 mm or less, it is asthough it is coming through a pinhole lens configuration, which meansthat each individual beamlet is always in relative focus on the retina,independent of the accommodation state of the eye—however the trajectoryof each beamlet will be affected by the accommodation state. Forinstance, if the beamlets approach the eye in parallel, representing adiscretized collimated aggregate wavefront, then an eye that iscorrectly accommodated to infinity will deflect the beamlets to allconverge upon the same shared spot on the retina, and will appear infocus. If the eye accommodates to, say, 1 m, the beams will be convergedto a spot in front of the retina, cross paths, and fall on multipleneighboring or partially overlapping spots on the retina—appearingblurred.

If the beamlets approach the eye in a diverging configuration, with ashared point of origin 1 meter from the viewer, then an accommodation of1 m will steer the beams to a single spot on the retina, and will appearin focus; if the viewer accommodates to infinity, the beamlets willconverge to a spot behind the retina, and produce multiple neighboringor partially overlapping spots on the retina, producing a blurred image.Stated more generally, the accommodation of the eye determines thedegree of overlap of the spots on the retina, and a given pixel is “infocus” when all of the spots are directed to the same spot on the retinaand “defocused” when the spots are offset from one another. This notionthat all of the 0.5 mm diameter or less beamlets are always in focus,and that they may be aggregated to be perceived by the eyes/brain asthough they are substantially the same as coherent wavefronts, may beutilized in producing configurations for comfortable three-dimensionalvirtual or augmented reality perception.

In other words, a set of multiple narrow beams may be used to emulatewhat is going on with a larger diameter variable focus beam, and if thebeamlet diameters are kept to a maximum of about 0.5 mm, then theymaintain a relatively static focus level, and to produce the perceptionof out-of-focus when desired, the beamlet angular trajectories may beselected to create an effect much like a larger out-of-focus beam (sucha defocussing treatment may not be the same as a Gaussian blur treatmentas for the larger beam, but will create a multimodal point spreadfunction that may be interpreted in a similar fashion to a Gaussianblur).

In a preferred embodiment, the beamlets are not mechanically deflectedto form this aggregate focus effect, but rather the eye receives asuperset of many beamlets that includes both a multiplicity of incidentangles and a multiplicity of locations at which the beamlets intersectthe pupil; to represent a given pixel from a particular viewingdistance, a subset of beamlets from the superset that comprise theappropriate angles of incidence and points of intersection with thepupil (as if they were being emitted from the same shared point oforigin in space) are turned on with matching color and intensity, torepresent that aggregate wavefront, while beamlets in the superset thatare inconsistent with the shared point of origin are not turned on withthat color and intensity (but some of them may be turned on with someother color and intensity level to represent, e.g., a different pixel).

Referring to FIG. 11A, each of a multiplicity of incoming beamlets (332)is passing through a small exit pupil (330) relative to the eye (58) ina discretized wavefront display configuration. Referring to FIG. 11B, asubset (334) of the group of beamlets (332) may be driven with matchingcolor and intensity levels to be perceived as though they are part ofthe same larger-sized ray (the bolded subgroup 334 may be deemed an“aggregated beam”). In this case, the subset of beamlets are parallel toone another, representing a collimated aggregate beam from opticalinfinity (such as light coming from a distant mountain). The eye isaccommodated to infinity, so the subset of beamlets are deflected by theeye's cornea and lens to all fall substantially upon the same locationof the retina and are perceived to comprise a single in focus pixel.

FIG. 11C shows another subset of beamlets representing an aggregatedcollimated beam (336) coming in from the right side of the field of viewof the user's eye (58) if the eye (58) is viewed in a coronal-styleplanar view from above. Again, the eye is shown accommodated toinfinity, so the beamlets fall on the same spot of the retina, and thepixel is perceived to be in focus. If, in contrast, a different subsetof beamlets were chosen that were reaching the eye as a diverging fan ofrays, those beamlets would not fall on the same location of the retina(and be perceived as in focus) until the eye were to shift accommodationto a near point that matches the geometrical point of origin of that fanof rays.

As regards patterns of points of intersection of beamlets with theanatomical pupil of the eye (i.e., the pattern of exit pupils), they maybe organized in configurations such as a cross-sectionally efficienthex-lattice (for example, as shown in FIG. 12A) or a square lattice orother two-dimensional array. Further, a three-dimensional array of exitpupils could be created, as well as time-varying arrays of exit pupils.

Discretized aggregate wavefronts may be created using severalconfigurations, such as an array of microdisplays or microprojectorsplaced optically conjugate with the exit pupil of viewing optics,microdisplay or microprojector arrays coupled to a direct field of viewsubstrate (such as an eyeglasses lens) such that they project light tothe eye directly, without additional intermediate viewing optics,successive spatial light modulation array techniques, or waveguidetechniques such as those described in relation to FIG. 8K.

Referring to FIG. 12A, in one embodiment, a lightfield may be created bybundling a group of small projectors or display units (such as scannedfiber displays). FIG. 12A depicts a hexagonal lattice projection bundle(338) which may, for example, create a 7 mm-diameter hex array with eachfiber display outputting a sub-image (340). If such an array has anoptical system, such as a lens, placed in front of it such that thearray is placed optically conjugate with the eye's entrance pupil, thiswill create an image of the array at the eye's pupil, as shown in FIG.12B, which essentially provides the same optical arrangement as theembodiment of FIG. 11A.

Each of the small exit pupils of the configuration is created by adedicated small display in the bundle (338), such as a scanning fiberdisplay. Optically, it's as though the entire hex array (338) ispositioned right into the anatomical pupil (45). Such embodiments aremeans for driving different subimages to different small exit pupilswithin the larger anatomical entrance pupil (45) of the eye, comprisinga superset of beamlets with a multiplicity of incident angles and pointsof intersection with the eye pupil. Each of the separate projectors ordisplays may be driven with a slightly different image, such thatsubimages may be created that pull out different sets of rays to bedriven at different light intensities and colors.

In one variation, a strict image conjugate may be created, as in theembodiment of FIG. 12B, wherein there is direct 1-to-1 mapping of thearray (338) with the pupil (45). In another variation, the spacing maybe changed between displays in the array and the optical system (lens342, in FIG. 12B) so that instead of getting a conjugate mapping of thearray to the eye pupil, the eye pupil may be catching the rays from thearray at some other distance. With such a configuration, one would stillget an angular diversity of beams through which one could create adiscretized aggregate wavefront representation, but the mathematicsregarding how to drive which ray and at which power and intensity maybecome more complex (although, on the other hand, such a configurationmay be considered simpler from a viewing optics perspective). Themathematics involved with light field image capture may be leveraged forthese calculations.

Referring to FIG. 13A, another lightfield creating embodiment isdepicted wherein an array of microdisplays or microprojectors (346) maybe coupled to a frame (344; such as an eyeglasses frame) to bepositioned in front of the eye (58). The depicted configuration is anonconjugate arrangement wherein there are no large-scale opticalelements interposed between the displays (for example, scanning fiberdisplays) of the array (346) and the eye (58). One can imagine a pair ofglasses, and coupled to those glasses are a plurality of displays, suchas scanning fiber engines, positioned orthogonal to the eyeglassessurface, and all angled inward so they are pointing at the pupil of theuser. Each display may be configured to create a set of raysrepresenting different elements of the beamlet superset.

With such a configuration, at the anatomical pupil (45) the user isgoing to receive a similar result as received in the embodimentsdiscussed in reference to FIG. 11A, in which every point at the user'spupil is receiving rays with a multiplicity of angles of incidence andpoints of intersection that are being contributed from the differentdisplays. FIG. 13B illustrates a nonconjugate configuration similar tothat of FIG. 13A, with the exception that the embodiment of FIG. 13Bfeatures a reflecting surface (348) to facilitate moving the displayarray (346) away from the eye's (58) field of view, while also allowingviews of the real world (144) through the reflective surface (348).

Thus another configuration for creating the angular diversity necessaryfor a discretized aggregate wavefront display is presented. To optimizesuch a configuration, the sizes of the displays may be decreased to themaximum. Scanning fiber displays which may be utilized as displays mayhave baseline diameters in the range of 1 mm, but reduction in enclosureand projection lens hardware may decrease the diameters of such displaysto about 0.5 mm or less, which is less disturbing for a user. Anotherdownsizing geometric refinement may be achieved by directly coupling acollimating lens (which may, for example, comprise a gradient refractiveindex, or “GRIN”, lens, a conventional curved lens, or a diffractivelens) to the tip of the scanning fiber itself in a case of a fiberscanning display array. For example, referring to FIG. 13D, a GRIN lens(354) is shown fused to the end of a single mode optical fiber. Anactuator (350; such as a piezoelectric actuator) is coupled to the fiber(352) and may be used to scan the fiber tip.

In another embodiment the end of the fiber may be shaped into ahemispherical shape using a curved polishing treatment of an opticalfiber to create a lensing effect. In another embodiment a standardrefractive lens may be coupled to the end of each optical fiber using anadhesive. In another embodiment a lens may be built from a dab oftransmissive polymeric material or glass, such as epoxy. In anotherembodiment the end of an optical fiber may be melted to create a curvedsurface for a lensing effect.

FIG. 13C-2 shows an embodiment wherein display configurations (i.e.,scanning fiber displays with GRIN lenses; shown in close-up view of FIG.13C-1) such as that shown in FIG. 13D may be coupled together through asingle transparent substrate (356) preferably having a refractive indexthat closely matches the cladding of the optical fibers (352) so thatthe fibers themselves are not very visible for viewing of the outsideworld across the depicted assembly (if the index matching of thecladding is done precisely, then the larger cladding/housing becomestransparent and only the tiny cores, which preferably are about 3microns in diameter, will be obstructing the view. In one embodiment thematrix (358) of displays may all be angled inward so they are directedtoward the anatomic pupil of the user (in another embodiment, they maystay parallel to each other, but such a configuration is lessefficient).

Referring to FIG. 13E, another embodiment is depicted wherein ratherthan using circular fibers to move cyclically, a thin series of planarwaveguides (358) are configured to be cantilevered relative to a largersubstrate structure (356). In one variation, the substrate (356) may bemoved to produce cyclic motion (i.e., at the resonant frequency of thecantilevered members 358) of the planar waveguides relative to thesubstrate structure. In another variation, the cantilevered waveguideportions (358) may be actuated with piezoelectric or other actuatorsrelative to the substrate. Image illumination information may beinjected, for example, from the right side (360) of the substratestructure to be coupled into the cantilevered waveguide portions (358).In one embodiment the substrate (356) may comprise a waveguideconfigured (such as with an integrated DOE configuration as describedabove) to totally internally reflect incoming light (360) along itslength and then redirect it to the cantilevered waveguide portions(358). As a person gazes toward the cantilevered waveguide portions(358) and through to the real world (144) behind, the planar waveguidesare configured to minimize any dispersion and/or focus changes withtheir planar shape factors.

In the context of discussing discretized aggregate wavefront displays,there is value placed in having some angular diversity created for everypoint in the exit pupil of the eye. In other words, it is desirable tohave multiple incoming beams to represent each pixel in a displayedimage. Referring to FIGS. 13F-1 and 13F-2, one way to gain furtherangular and spatial diversity is to use a multicore fiber and place alens at the exit point, such as a GRIN lens, so that the exit beams aredeflected through a single nodal point (366); that nodal point may thenbe scanned back and forth in a scanned fiber type of arrangement (suchas by a piezoelectric actuator 368). If a retinal conjugate is placed atthe plane defined at the end of the GRIN lens, a display may be createdthat is functionally equivalent to the general case discretizedaggregate wavefront configuration described above.

Referring to FIG. 13G, a similar effect may be achieved not by using alens, but by scanning the face of a multicore system at the correctconjugate of an optical system (372), the goal being to create a higherangular and spatial diversity of beams. In other words, rather thanhaving a bunch of separately scanned fiber displays as in the bundledexample of FIG. 12A described above, some of this requisite angular andspatial diversity may be created through the use of multiple cores tocreate a plane which may be relayed by a waveguide. Referring to FIG.13H, a multicore fiber (362) may be scanned (such as by a piezoelectricactuator 368) to create a set of beamlets with a multiplicity of anglesof incidence and points of intersection which may be relayed to the eye(58) by a waveguide (370). Thus in one embodiment a collimatedlightfield image may be injected into a waveguide, and without anyadditional refocusing elements, that lightfield display may betranslated directly to the human eye.

FIGS. 13I-13L depict certain commercially available multicore fiber(362) configurations (from vendors such as Mitsubishi Cable Industries,Ltd. of Japan), including one variation (363) with a rectangular crosssection, as well as variations with flat exit faces (372) and angledexit faces (374).

Referring to FIG. 13M, some additional angular diversity may be createdby having a waveguide (376) fed with a linear array of displays (378),such as scanning fiber displays.

Referring to FIGS. 14A-14F, another group of configurations for creatinga fixed viewpoint lightfield display is described. Referring back toFIG. 11A, if a two-dimensional plane was created that was intersectingall of the tiny beams coming in from the left, each beamlet would have acertain point of intersection with that plane. If another plane wascreated at a different distance to the left, then all of the beamletswould intersect that plane at a different location. Then going back toFIG. 14A, if various positions on each of two or more planes can beallowed to selectively transmit or block the light radiation directedthrough it, such a multi-planar configuration may be utilized toselectively create a lightfield by independently modulating individualbeamlets.

The basic embodiment of FIG. 14A shows two spatial light modulators,such as liquid crystal display panels (380, 382; in other embodimentsthey may be MEMS shutter displays or DLP DMD arrays) which may beindependently controlled to block or transmit different rays on ahigh-resolution basis. For example, referring to FIG. 14A, if the secondpanel (382) blocks or attenuates transmission of rays at point “a”(384), all of the depicted rays will be blocked; but if only the firstpanel (380) blocks or attenuates transmission of rays at point “b”(386), then only the lower incoming ray (388) will beblocked/attenuated, while the rest will be transmitted toward the pupil(45). Each of the controllable panels or planes may be deemed a “spatiallight modulator” or “fatte”. The intensity of each transmitted beampassed through a series of SLMs will be a function of the combination ofthe transparency of the various pixels in the various SLM arrays. Thuswithout any sort of lens elements, a set of beamlets with a multiplicityof angles and points of intersection (or a “lightfield”) may be createdusing a plurality of stacked SLMs. Additional numbers of SLMs beyond twoprovides more opportunities to control which beams are selectivelyattenuated.

As noted briefly above, in addition to using stacked liquid crystaldisplays as SLMs, planes of DMD devices from DLP systems may be stackedto function as SLMs, and may be preferred over liquid crystal systems asSLMs due to their ability to more efficiently pass light (with a mirrorelement in a first state, reflectivity to the next element on the way tothe eye may be quite efficient; with a mirror element in a second state,the mirror angle may be moved by an angle such as 12 degrees to directthe light away from the path to the eye). Referring to FIG. 14B, in oneDMD embodiment, two DMDs (390, 390) may be utilized in series with apair of lenses (394, 396) in a periscope type of configuration tomaintain a high amount of transmission of light from the real world(144) to the eye (58) of the user. The embodiment of FIG. 14C providessix different DMD (402, 404, 406, 408, 410, 412) plane opportunities tointercede from an SLM functionality as beams are routed to the eye (58),along with two lenses (398, 400) for beam control.

FIG. 14D illustrates a more complicated periscope type arrangement withup to four DMDs (422, 424, 426, 428) for SLM functionality and fourlenses (414, 420, 416, 418); this configuration is designed to ensurethat the image does not become flipped upside down as it travels throughto the eye (58). FIG. 14E illustrates in embodiment wherein light may bereflected between two different DMD devices (430, 432) without anyintervening lenses (the lenses in the above designs are useful in suchconfigurations for incorporating image information from the real world),in a hall-of-mirrors type of arrangement wherein the display may beviewed through the “hall of mirrors” and operates in a modesubstantially similar to that illustrated in FIG. 14A. FIG. 14Fillustrates an embodiment wherein a the non-display portions of twofacing DMD chips (434, 436) may be covered with a reflective layer topropagate light to and from active display regions (438, 440) of the DMDchips. In other embodiments, in place of DMDs for SLM functionality,arrays of sliding MEMS shutters (such as those available from vendorssuch as Pixtronics, a division of Qualcomm, Inc.) may be utilized toeither pass or block light. In another embodiment, arrays of smalllouvers that move out of place to present light-transmitting aperturesmay similarly be aggregated for SLM functionality.

A lightfield of many small beamlets (say, less than about 0.5 mm indiameter) may be injected into and propagated through a waveguide orother optical system. For example, a conventional “birdbath” type ofoptical system may be suitable for transferring the light of alightfield input, or a freeform optics design, as described below, orany number of waveguide configurations. FIGS. 15A-15C illustrate the useof a wedge type waveguide (442) along with a plurality of light sourcesas another configuration useful in creating a lightfield. Referring toFIG. 15A, light may be injected into the wedge-shaped waveguide (442)from two different locations/displays (444, 446), and will emergeaccording to the total internal reflection properties of thewedge-shaped waveguide at different angles (448) based upon the pointsof injection into the waveguide.

Referring to FIG. 15B, if one creates a linear array (450) of displays(such as scanning fiber displays) projecting into the end of thewaveguide as shown, then a large angular diversity of beams (452) willbe exiting the waveguide in one dimension, as shown in FIG. 15C. Indeed,if one contemplates adding yet another linear array of displaysinjecting into the end of the waveguide but at a slightly differentangle, then an angular diversity of beams may be created that exitssimilarly to the fanned out exit pattern shown in FIG. 15C, but at anorthogonal axis; together these may be utilized to create atwo-dimensional fan of rays exiting each location of the waveguide. Thusanother configuration is presented for creating angular diversity toform a lightfield display using one or more scanning fiber displayarrays (or alternatively using other displays which will meet the spacerequirements, such as miniaturized DLP projection configurations).

Alternatively, as an input to the wedge-shaped waveguides shown herein,a stack of SLM devices may be utilized, in which case rather than thedirect view of SLM output as described above, the lightfield output fromthe SLM configuration may be used as an input to a configuration such asthat shown in FIG. 15C. One of the key concepts here is that while aconventional waveguide is best suited to relay beams of collimated lightsuccessfully, with a lightfield of small-diameter collimated beams,conventional waveguide technology may be utilized to further manipulatethe output of such a lightfield system as injected into the side of awaveguide, such as a wedge-shaped waveguide, due to the beamsize/collimation.

In another related embodiment, rather than projecting with multipleseparate displays, a multicore fiber may be used to generate alightfield and inject it into the waveguide. Further, a time-varyinglightfield may be utilized as an input, such that rather than creating astatic distribution of beamlets coming out of a lightfield, one may havesome dynamic elements that are methodically changing the path of the setof beams. They may be done using components such as waveguides withembedded DOEs (e.g., such as those described above in reference to FIGS.8B-8N, or liquid crystal layers, as described in reference to FIG. 7B),wherein two optical paths are created (one smaller total internalreflection path wherein a liquid crystal layer is placed in a firstvoltage state to have a refractive index mismatch with the othersubstrate material that causes total internal reflection down just theother substrate material's waveguide; one larger total internalreflection optical path wherein the liquid crystal layer is placed in asecond voltage state to have a matching refractive index with the othersubstrate material, so that the light totally internally reflectsthrough the composite waveguide which includes both the liquid crystalportion and the other substrate portion). Similarly a wedge-shapedwaveguide may be configured to have a bi-modal total internal reflectionparadigm (for example, in one variation, wedge-shaped elements may beconfigured such that when a liquid crystal portion is activated, notonly is the spacing changed, but also the angle at which the beams arereflected).

One embodiment of a scanning light display may be characterized simplyas a scanning fiber display with a lens at the end of the scanned fiber.Many lens varieties are suitable, such as a GRIN lens, which may be usedto collimate the light or to focus the light down to a spot smaller thanthe fiber's mode field diameter providing the advantage of producing anumerical aperture (or “NA”) increase and circumventing the opticalinvariant, which is correlated inversely with spot size. Smaller spotsize generally facilitates a higher resolution opportunity from adisplay perspective, which generally is preferred. In one embodiment, aGRIN lens may be long enough relative to the fiber that it may comprisethe vibrating element (i.e., rather than the usual distal fiber tipvibration with a scanned fiber display)—a configuration which may bedeemed a “scanned GRIN lens display”.

In another embodiment, a diffractive lens may be utilized at the exitend of a scanning fiber display (i.e., patterned onto the fiber). Inanother embodiment, a curved mirror may be positioned on the end of thefiber that operates in a reflecting configuration. Essentially any ofthe configurations known to collimate and focus a beam may be used atthe end of a scanning fiber to produce a suitable scanned light display.

Two significant utilities to having a lens coupled to or comprising theend of a scanned fiber (i.e., as compared to configurations wherein anuncoupled lens may be utilized to direct light after it exits a fiber)are a) the light exiting may be collimated to obviate the need to useother external optics to do so; b) the NA, or the angle of the cone atwhich light sprays out the end of the single-mode fiber core, may beincreased, thereby decreasing the associated spot size for the fiber andincreasing the available resolution for the display.

As described above, a lens such as a GRIN lens may be fused to orotherwise coupled to the end of an optical fiber or formed from aportion of the end of the fiber using techniques such as polishing. Inone embodiment, a typical optical fiber with an NA of about 0.13 or 0.14may have a spot size (also known as the “mode field diameter” for theoptical fiber given the NA) of about 3 microns. This provides forrelatively high resolution display possibilities given the industrystandard display resolution paradigms (for example, a typicalmicrodisplay technology such as LCD or organic light emitting diode, or“OLED” has a spot size of about 5 microns). Thus the aforementionedscanning light display may have ⅗ of the smallest pixel pitch availablewith a conventional display; further, using a lens at the end of thefiber, the aforementioned configuration may produce a spot size in therange of 1-2 microns.

In another embodiment, rather than using a scanned cylindrical fiber, acantilevered portion of a waveguide (such as a waveguide created usingmicrofabrication processes such as masking and etching, rather thandrawn microfiber techniques) may be placed into scanning oscillatorymotion, and may be fitted with lensing at the exit ends.

In another embodiment, an increased numerical aperture for a fiber to bescanned may be created using a diffuser (i.e., one configured to scatterlight and create a larger NA) covering the exit end of the fiber. In onevariation, the diffuser may be created by etching the end of the fiberto create small bits of terrain that scatter light; in another variationa bead or sandblasting technique, or direct sanding/scuffing techniquemay be utilized to create scattering terrain. In another variation, anengineered diffuser, similar to a diffractive element, may be created tomaintain a clean spot size with desirable NA, which ties into the notionof using a diffractive lens, as noted above.

Referring to FIG. 16A, an array of optical fibers (454) is shown coupledin to a coupler (456) configured to hold them in parallel together sothat their ends may be ground and polished to have an output edge at acritical angle (458; 42 degrees for most glass, for example) to thelongitudinal axes of the input fibers, such that the light exiting theangled faces will exit as though it had been passing through a prism,and will bend and become nearly parallel to the surfaces of the polishedfaces. The beams exiting the fibers (454) in the bundle will becomesuperimposed, but will be out of phase longitudinally due to thedifferent path lengths (referring to FIG. 16B, for example, thedifference in path lengths from angled exit face to focusing lens forthe different cores is visible).

What was an X axis type of separation in the bundle before exit from theangled faces, will become a Z axis separation, a fact that is helpful increating a multifocal light source from such a configuration. In anotherembodiment, rather than using a bundled/coupled plurality of single modefibers, a multicore fiber, such as those available from Mitsubishi CableIndustries, Ltd. of Japan, may be angle polished.

In one embodiment, if a 45 degree angle is polished into a fiber andthen covered with a reflective element, such as a mirror coating, theexiting light may be reflected from the polished surface and emerge fromthe side of the fiber (in one embodiment at a location wherein aflat-polished exit window has been created in the side of the fiber)such that as the fiber is scanned in what would normally be an X-YCartesian coordinate system axis, that fiber would now be functionallyperforming the equivalent of an X-Z scan, with the distance changingduring the course of the scan. Such a configuration may be beneficiallyutilized to change the focus of the display as well.

Multicore fibers may be configured to play a role in display resolutionenhancement (i.e., higher resolution). For example, in one embodiment,if separate pixel data is sent down a tight bundle of 19 cores in amulticore fiber, and that cluster is scanned around in a sparse spiralpattern with the pitch of the spiral being approximately equal to thediameter of the multicore, then sweeping around will effectively createa display resolution that is approximately 19× the resolution of asingle core fiber being similarly scanned around. Indeed, it may be morepractical to have the fibers more sparsely positioned relative to eachother, as in the configuration of FIG. 16C, which has 7 clusters (464; 7is used for illustrative purposes because it is an efficient tiling/hexpattern; other patterns or numbers may be utilized; for example, acluster of 19; the configuration is scalable up or down) of 3 fiberseach housed within a conduit (462).

With a sparse configuration as shown in FIG. 16C, scanning of themulticore scans each of the cores through its own local region, asopposed to a configuration wherein the cores are all packed tightlytogether and scanned (wherein cores end up overlapping with scanning; ifthe cores are too close to each other, the NA of the core is not largeenough and the very closely packed cores end up blurring togethersomewhat and not creating as discriminable a spot for display). Thus,for resolution increases, it is preferable to have sparse tiling ratherthan highly dense tiling, although both will work.

The notion that densely packed scanned cores can create blurring at thedisplay may be utilized as an advantage in one embodiment wherein aplurality (say a triad or cores to carry red, green, and blue light) ofcores may be intentionally packed together densely so that each triadforms a triad of overlapped spots featuring red, green, and blue light.With such a configuration, one is able to have an RGB display withouthaving to combine red, green, and blue into a single-mode core, which isan advantage, because conventional mechanisms for combining a plurality(such as three) wavelets of light into a single core are subject tosignificant losses in optical energy. Referring to FIG. 16C, in oneembodiment each tight cluster of 3 fiber cores contains one core thatrelays red light, one core that relays green light, and one core thatrelays blue light, with the 3 fiber cores close enough together thattheir positional differences are not resolvable by the subsequent relayoptics, forming an effectively superimposed RGB pixel; thus, the sparsetiling of 7 clusters produces resolution enhancement while the tightpacking of 3 cores within the clusters facilitates seamless colorblending without the need to utilize glossy RGB fiber combiners (e.g.,those using wavelength division multiplexing or evanescent couplingtechniques).

Referring to FIG. 16D, in another more simple variation, one may havejust one cluster (464) housed in a conduit (468) for, say,red/green/blue (and in another embodiment, another core may be added forinfrared for uses such as eye tracking). In another embodiment,additional cores may be placed in the tight cluster to carryingadditional wavelengths of light to comprise a multi-primary display forincreased color gamut. Referring to FIG. 16E, in another embodiment, asparse array of single cores (470); in one variation with red, green,and blue combined down each of them) within a conduit (466) may beutilized; such a configuration is workable albeit somewhat lessefficient for resolution increase, but not optimum for red/green/bluecombining.

Multicore fibers also may be utilized for creating lightfield displays.Indeed, rather than keeping the cores separated enough from each otherso that the cores do not scan on each other's local area at the displaypanel, as described above in the context of creating a scanning lightdisplay, with a lightfield display, it is desirable to scan around adensely packed plurality of fibers because each of the beams producedrepresents a specific part of the lightfield. The light exiting from thebundled fiber tips can be relatively narrow if the fibers have a smallNA; lightfield configurations may take advantage of this and have anarrangement in which at the anatomic pupil, a plurality of slightlydifferent beams are being received from the array. Thus there areoptical configurations with scanning a multicore that are functionallyequivalent to an array of single scanning fiber modules, and thus alightfield may be created by scanning a multicore rather than scanning agroup of single mode fibers.

In one embodiment, a multi-core phased array approach may be used tocreate a large exit pupil variable wavefront configuration to facilitatethree-dimensional perception. A single laser configuration with phasemodulators is described above. In a multicore embodiment, phase delaysmay be induced into different channels of a multicore fiber, such that asingle laser's light is injected into all of the cores of the multicoreconfiguration so that there is mutual coherence.

In one embodiment, a multi-core fiber may be combined with a lens, suchas a GRIN lens. Such lens may be, for example, a refractive lens,diffractive lens, or a polished edge functioning as a lens. The lens maybe a single optical surface, or may comprise multiple optical surfacesstacked up. Indeed, in addition to having a single lens that extends thediameter of the multicore, a smaller lenslet array may be desirable atthe exit point of light from the cores of the multicore, for example.FIG. 16F shows an embodiment wherein a multicore fiber (470) is emittingmultiple beams into a lens (472), such as a GRIN lens. The lens collectsthe beams down to a focal point (474) in space in front of the lens. Inmany conventional configurations, the beams would exit the multicorefiber as diverging. The GRIN or other lens is configured to function todirect them down to a single point and collimate them, such that thecollimated result may be scanned around for a lightfield display, forinstance.

Referring to FIG. 16G, smaller lenses (478) may be placed in front ofeach of the cores of a multicore (476) configuration, and these lensesmay be utilized to collimate; then a shared lens (480) may be configuredto focus the collimated beams down to a diffraction limited spot (482)that is aligned for all of the three spots. The net result of such aconfiguration: by combining three collimated, narrow beams with narrowNA together as shown, one effectively combines all three into a muchlarger angle of emission which translates to a smaller spot size in, forexample, a head mounted optical display system which may be next in thechain of light delivery to the user.

Referring to FIG. 16H, one embodiment features a multicore fiber (476)with a lenslet (478) array feeding the light to a small prism array(484) that deflects the beams generated by the individual cores to acommon point. Alternatively one may have the small lenslet array shiftedrelative to the cores such that the light is being deflected and focuseddown to a single point. Such a configuration may be utilized to increasethe numerical aperture.

Referring to FIG. 16I, a two-step configuration is shown with a smalllenslet (478) array capturing light from the multicore fiber (476),followed sequentially by a shared lens (486) to focus the beams to asingle point (488). Such a configuration may be utilized to increase thenumerical aperture. As discussed above, a larger NA corresponds to asmaller pixel size and higher possible display resolution.

Referring to FIG. 16J, a beveled fiber array which may be held togetherwith a coupler (456), such as those described above, may be scanned witha reflecting device (494; such as a DMD module of a DLP system). Withmultiple single fibers (454) coupled into the array, or a multicoreinstead, the superimposed light can be directed through one or morefocusing lenses (490, 492) to create a multifocal beam; with thesuperimposing and angulation of the array, the different sources aredifferent distances from the focusing lens, which creates differentfocus levels in the beams as they emerge from the lens (492) and aredirected toward the retina (54) of the eye (58) of the user. Forexample, the farthest optical route/beam may be set up to be acollimated beam representative of optical infinity focal positions.Closer routes/beams may be associated with diverging sphericalwavefronts of closer focal locations.

The multifocal beam may be passed into a scanning mirror which may beconfigured to create a raster scan (or, for example, a Lissajous curvescan pattern or a spiral scan pattern) of the multifocal beam which maybe passed through a series of focusing lenses and then to the cornea andcrystalline lens of the eye. The various beams emerging from the lensesare creating different pixels or voxels of varying focal distances thatare superimposed.

In one embodiment, one may write different data to each of the lightmodulation channels at the front end, thereby creating an image that isprojected to the eye with one or more focus elements. By changing thefocal distance of the crystalline lens (i.e., by accommodating), theuser can bring different incoming pixels into and out of focus, as shownin FIGS. 16K and 16L wherein the crystalline lens is in different Z axispositions. In another embodiment, the fiber array may be actuated/movedaround by a piezoelectric actuator. In another embodiment, a relativelythin ribbon array may be resonated in cantilevered form along the axisperpendicular to the arrangement of the array fibers (i.e., in the thindirection of the ribbon) when a piezoelectric actuator is activated. Inone variation, a separate piezoelectric actuator may be utilized tocreate a vibratory scan in the orthogonal long axis. In anotherembodiment, a single mirror axis scan may be employed for a slow scanalong the long axis while the fiber ribbon is vibrated resonantly.

Referring to FIG. 16M, an array (496) of scanning fiber displays (498)may be beneficially bundled/tiled for an effective resolution increase,the notion being that with such as configuration, each scanning fiber ofthe bundle is configured to write to a different portion of the imageplane (500), as shown, for example, in FIG. 16N, wherein each portion ofthe image plane is addressed by the emissions from a least one bundle.In other embodiments, optical configurations may be utilized that allowfor slight magnification of the beams as they exit the optical fiber sothat there is some overlap in the hexagonal, or other lattice pattern,that hits the display plane, so there is a better fill factor while alsomaintaining an adequately small spot size in the image plane andunderstanding that there is a subtle magnification in that image plane.

Rather than having individual lenses at the end of each scanned fiberenclosure housing, in one embodiment a monolithic lenslet array may beutilized, so that the lenses can be as closely packed as possible, whichallows for even smaller spot sizes in the image plane because one mayuse a lower amount of magnification in the optical system. Thus arraysof fiber scan displays may be used to increase the resolution of thedisplay, or in other words, they may be used to increase the field ofview of the display, because each engine is being used to scan adifferent portion of the field of view.

For a lightfield configuration, the emissions may be more desirablyoverlapped at the image plane. In one embodiment, a lightfield displaymay be created using a plurality of small diameter fibers scanned aroundin space. For example, instead of having all of the fibers address adifferent part of an image plane as described above, have moreoverlapping, more fibers angled inward, etc., or change the focal powerof the lenses so that the small spot sizes are not conjugate with atiled image plane configuration. Such a configuration may be used tocreate a lightfield display to scan lots of smaller diameter rays aroundthat become intercepted in the same physical space.

Referring back to FIG. 12B, it was discussed that one way of creating alightfield display involves making the output of the elements on theleft collimated with narrow beams, and then making the projecting arrayconjugate with the eye pupil on the right.

Referring to FIG. 16O, with a common substrate block (502), a singleactuator may be utilized to actuate a plurality of fibers (506) inunison together. A similar configuration is discussed above in referenceto FIGS. 13-C-1 and 13-C-2. It may be practically difficult to have allof the fibers retain the same resonant frequency, vibrate in a desirablephase relationship to each other, or have the same dimensions ofcantilevering from the substrate block. To address this challenge, thetips of the fibers may be mechanically coupled with a lattice or sheet(504), such as a graphene sheet that is very thin, rigid, and light inweight. With such a coupling, the entire array may vibrate similarly andhave the same phase relationship. In another embodiment a matrix ofcarbon nanotubes may be utilized to couple the fibers, or a piece ofvery thin planar glass (such as the kind used in creating liquid crystaldisplay panels) may be coupled to the fiber ends. Further, a laser orother precision cutting device may be utilized to cut all associatedfibers to the same cantilevered length.

Referring to FIG. 17, in one embodiment it may be desirable to have acontact lens directly interfaced with the cornea, and configured tofacilitate the eye focusing on a display that is quite close (such asthe typical distance between a cornea and an eyeglasses lens). Ratherthan placing an optical lens as a contact lens, in one variation thelens may comprise a selective filter. FIG. 17 depicts a plot (508) whatmay be deemed a “notch filter”, due to its design to block only certainwavelength bands, such as 450 nm (peak blue), 530 nm (green), and 650nm, and generally pass or transmit other wavelengths. In one embodimentseveral layers of dielectric coatings may be aggregated to provide thenotch filtering functionality.

Such a filtering configuration may be coupled with a scanning fiberdisplay that is producing a very narrow band illumination for red,green, and blue, and the contact lens with the notch filtering willblock out all of the light coming from the display (such as aminidisplay, such as an OLED display, mounted in a position normallyoccupied by an eyeglasses lens) except for the transmissive wavelengths.A narrow pinhole may be created in the middle of the contact lensfiltering layers/film such that the small aperture (i.e., less thanabout 1.5 mm diameter) does allow passage of the otherwise blockedwavelengths. Thus a pinhole lens configuration is created that functionsin a pinhole manner for red, green, and blue only to intake images fromthe minidisplay, while light from the real world, which generally isbroadband illumination, will pass through the contact lens relativelyunimpeded. Thus a large depth of focus virtual display configuration maybe assembled and operated. In another embodiment, a collimated imageexiting from a waveguide would be visible at the retina because of thepinhole large-depth-of-focus configuration.

It may be useful to create a display that can vary its depth of focusover time. For example, in one embodiment, a display may be configuredto have different display modes that may be selected (preferably rapidlytoggling between the two at the command of the operator) by an operator,such as a first mode combining a very large depth of focus with a smallexit pupil diameter (i.e., so that everything is in focus all of thetime), and a second mode featuring a larger exit pupil and a more narrowdepth of focus. In operation, if a user is to play a three-dimensionalvideo game with objects to be perceived at many depths of field, theoperator may select the first mode; alternatively, if a user is to typein a long essay (i.e., for a relatively long period of time) using atwo-dimensional word processing display configuration, it may be moredesirable to switch to the second mode to have the convenience of alarger exit pupil, and a sharper image.

In another embodiment, it may be desirable to have a multi-depth offocus display configuration wherein some subimages are presented with alarge depth of focus while other subimages are presented with smalldepth of focus. For example, one configuration may have red wavelengthand blue wavelength channels presented with a very small exit pupil sothat they are always in focus. Then, a green channel only may bepresented with a large exit pupil configuration with multiple depthplanes (i.e., because the human accommodation system tends topreferentially target green wavelengths for optimizing focus level).Thus, in order to cut costs associated with having too many elements torepresent with full depth planes in red, green, and blue, the greenwavelength may be prioritized and represented with various differentwavefront levels. Red and blue may be relegated to being representedwith a more Maxwellian approach (and, as described above in reference toMaxwellian displays, software may be utilized to induce Gaussian levelsof blur). Such a display would simultaneously present multiple depths offocus.

As described above, there are portions of the retina which have a higherdensity of light sensors. The fovea portion, for example, generally ispopulated with approximately 120 cones per visual degree. Displaysystems have been created in the past that use eye or gaze tracking asan input, and to save computation resources by only creating really highresolution rendering for where the person is gazing at the time, whilelower resolution rendering is presented to the rest of the retina; thelocations of the high versus low resolution portions may be dynamicallyslaved to the tracked gaze location in such a configuration, which maybe termed a “foveated display”.

An improvement on such configurations may comprise a scanning fiberdisplay with pattern spacing that may be dynamically slaved to trackedeye gaze. For example, with a typical scanning fiber display operatingin a spiral pattern, as shown in FIG. 18 (the leftmost portion 510 ofthe image in FIG. 18 illustrates a spiral motion pattern of a scannedmulticore fiber 514; the rightmost portion 512 of the image in FIG. 18illustrates a spiral motion pattern of a scanned single fiber 516 forcomparison), a constant pattern pitch provides for a uniform displayresolution.

In a foveated display configuration, a non-uniform scanning pitch may beutilized, with smaller/tighter pitch (and therefore higher resolution)dynamically slaved to the detected gaze location. For example, if theuser's gaze was detected as moving toward the edge of the displayscreen, the spirals may be clustered more densely in such location,which would create a toroid-type scanning pattern for thehigh-resolution portions, and the rest of the display being in alower-resolution mode. In a configuration wherein gaps may be created inthe portions of the display in a lower-resolution mode, blur could beintentionally dynamically created to smooth out the transitions betweenscans, as well as between transitions from high-resolution tolower-resolution scan pitch.

The term lightfield may be used to describe a volumetric 3-Drepresentation of light traveling from an object to a viewer's eye.However, an optical see-through display can only reflect light to theeye, not the absence of light, and ambient light from the real worldwill add to any light representing a virtual object. That is, if avirtual object presented to the eye contains a black or very darkportion, the ambient light from the real world may pass through thatdark portion and obscure that it was intended to be dark.

It is nonetheless desirable to be able to present a dark virtual objectover a bright real background, and for that dark virtual object toappear to occupy a volume at a desired viewing distance; i.e., it isuseful to create a “darkfield” representation of that dark virtualobject, in which the absence of light is perceived to be located at aparticular point in space. With regard to occlusion elements and thepresentation of information to the eye of the user so that he or she canperceive darkfield aspects of virtual objects, even in well lightedactual environments, certain aspects of the aforementioned spatial lightmodulator, or “SLM”, configurations are pertinent. As described above,with a light-sensing system such as the eye, one way to get selectiveperception of dark field to selectively attenuate light from suchportions of the display, because the subject display systems are aboutmanipulation and presentation of light; in other words, darkfield cannotbe specifically projected—it's the lack of illumination that may beperceived as darkfield, and thus, configurations for selectiveattenuation of illumination have been developed.

Referring back to the discussion of SLM configurations, one way toselectively attenuate for a darkfield perception is to block all of thelight coming from one angle, while allowing light from other angles tobe transmitted. This may be accomplished with a plurality of SLM planescomprising elements such as liquid crystal (which may not be the mostoptimal due to its relatively low transparency when in the transmittingstate), DMD elements of DLP systems (which have relative hightransmission/reflection ratios when in such mode), and MEMS arrays orshutters that are configured to controllably shutter or pass lightradiation, as described above.

With regard to suitable liquid crystal display (“LCD”) configurations, acholesteric LCD array may be utilized for a controlledocclusion/blocking array. As opposed to the conventional LCD paradigmwherein a polarization state is changed as a function of voltage, with acholesteric LCD configuration, a pigment is being bound to the liquidcrystal molecule, and then the molecule is physically tilted in responseto an applied voltage. Such a configuration may be designed to achievegreater transparency when in a transmissive mode than conventional LCD,and a stack of polarizing films is not needed as it is with conventionalLCD.

In another embodiment, a plurality of layers of controllably interruptedpatterns may be utilized to controllably block selected presentation oflight using moire effects. For example, in one configuration, two arraysof attenuation patterns, each of which may comprise, for example,fine-pitched sine waves printed or painted upon a transparent planarmaterial such as a glass substrate, may be presented to the eye of auser at a distance close enough that when the viewer looks througheither of the patterns alone, the view is essentially transparent, butif the viewer looks through both patterns lined up in sequence, theviewer will see a spatial beat frequency moire attenuation pattern, evenwhen the two attenuation patterns are placed in sequence relativelyclose to the eye of the user.

The beat frequency is dependent upon the pitch of the patterns on thetwo attenuation planes, so in one embodiment, an attenuation pattern forselectively blocking certain light transmission for darkfield perceptionmay be created using two sequential patterns, each of which otherwisewould be transparent to the user, but which together in series create aspatial beat frequency moire attenuation pattern selected to attenuatein accordance with the darkfield perception desired in the augmentedreality system.

In another embodiment a controlled occlusion paradigm for darkfieldeffect may be created using a multi-view display style occluder. Forexample, one configuration may comprise one pin-holed layer that fullyoccludes with the exception of small apertures or pinholes, along with aselective attenuation layer in series, which may comprise an LCD, DLPsystem, or other selective attenuation layer configuration, such asthose described above. In one scenario, with the pinhole array placed ata typical eyeglasses lens distance from the cornea (about 30 mm), andwith a selective attenuation panel located opposite the pinhole arrayfrom the eye, a perception of a sharp mechanical edge out in space maybe created. In essence, if the configuration will allow certain anglesof light to pass, and others to be blocked or occluded, than aperception of a very sharp pattern, such as a sharp edge projection, maybe created. In another related embodiment, the pinhole array layer maybe replaced with a second dynamic attenuation layer to provide asomewhat similar configuration, but with more controls than the staticpinhole array layer (the static pinhole layer could be simulated, butneed not be).

In another related embodiment, the pinholes may be replaced withcylindrical lenses. The same pattern of occlusion as in the pinholearray layer configuration may be achieved, but with cylindrical lenses,the array is not restricted to the very tiny pinhole geometries. Toprevent the eye from being presented with distortions due to the lenseswhen viewing through to the real world, a second lens array may be addedon the side of the aperture or lens array opposite of the side nearestthe eye to compensate and provide the view-through illumination withbasically a zero power telescope configuration.

In another embodiment, rather than physically blocking light forocclusion and creation of darkfield perception, the light may be bent orbounced, or a polarization of the light may be changed if a liquidcrystal layer is utilized. For example, in one variation, each liquidcrystal layer may act as a polarization rotator such that if a patternedpolarizing material is incorporated on one face of a panel, then thepolarization of individual rays coming from the real world may beselectively manipulated so they catch a portion of the patternedpolarizer. There are polarizers known in the art that have checkerboardpatterns wherein half of the “checker boxes” have vertical polarizationand the other half have horizontal polarization. In addition, if amaterial such as liquid crystal is used in which polarization may beselectively manipulated, light may be selectively attenuated with this.

As described above, selective reflectors may provide greatertransmission efficiency than LCD. In one embodiment, if a lens system isplaced such that it takes light coming in from the real world andfocuses a plane from the real world onto an image plane, and if a DMD(i.e., DLP technology) is placed at that image plane to reflect lightwhen in an “on” state towards another set of lenses that pass the lightto the eye, and those lenses also have the DMD at their focal length,the one may create an attenuation pattern that is in focus for the eye.In other words, DMDs may be used in a selective reflector plane in azero magnification telescope configuration, such as is shown in FIG.19A, to controllably occlude and facilitate creating darkfieldperception.

As shown in FIG. 19A, a lens (518) is taking light from the real world(144) and focusing it down to an image plane (520); if a DMD (or otherspatial attenuation device) (522) is placed at the focal length of thelens (i.e., at the image plane 520), the lens (518) is going to takewhatever light is coming from optical infinity and focus that onto theimage plane (520). Then the spatial attenuator (522) may be utilized toselectively block out things that are to be attenuated. FIG. 19A showsthe attenuator DMDs in the transmissive mode wherein they pass the beamsshown crossing the device. The image is then placed at the focal lengthof the second lens (524). Preferably the two lenses (518, 524) have thesame focal power so they end up being a zero-power telescope, or a“relay”, that does not magnify views to the real world (144). Such aconfiguration may be used to present unmagnified views of the worldwhile also allowing selective blocking/attenuation of certain pixels.

In another embodiment, as shown in FIGS. 19B and 19C, additional DMDsmay be added such that light reflects from each of four DMDs (526, 528,530, 532) before passing to the eye. FIG. 19B shows an embodiment withtwo lenses preferably with the same focal power (focal length “F”)placed at a 2F relationship from one another (the focal length of thefirst being conjugate to the focal length of the second) to have thezero-power telescope effect; FIG. 19C shows an embodiment withoutlenses. The angles of orientation of the four reflective panels (526,528, 530, 532) in the depicted embodiments of FIGS. 19B and 19C areshown to be around 45 degrees for simple illustration purposes, butspecific relative orientation is required (for example, a typical DMDreflect at about a 12 degree angle).

In another embodiment, the panels may also be ferroelectric, or may beany other kind of reflective or selective attenuator panel or array. Inone embodiment similar to those depicted in FIGS. 19B and 19C, one ofthe three reflector arrays may be a simple mirror, such that the other 3are selective attenuators, thus still providing three independent planesto controllably occlude portions of the incoming illumination infurtherance of darkfield perception. By having multiple dynamicreflective attenuators in series, masks at different optical distancesrelative to the real world may be created.

Alternatively, referring back to FIG. 19C, one may create aconfiguration wherein one or more DMDs are placed in a reflectiveperiscope configuration without any lenses. Such a configuration may bedriven in lightfield algorithms to selectively attenuate certain rayswhile others are passed.

In another embodiment, a DMD or similar matrix of controllably movabledevices may be created upon a transparent substrate as opposed to agenerally opaque substrate, for use in a transmissive configuration suchas virtual reality.

In another embodiment, two LCD panels may be utilized as lightfieldoccluders. In one variation, they may be thought of as attenuators dueto their attenuating capability as described above; alternatively theymay be considered polarization rotators with a shared polarizer stack.Suitable LCDs may comprise components such as blue phase liquid crystal,cholesteric liquid crystal, ferroelectric liquid crystal, and/or twistednematic liquid crystal.

One embodiment may comprise an array of directionally-selectiveocclusion elements, such as a MEMS device featuring a set of louversthat can change rotation such that they pass the majority of light thatis coming from a particular angle, but are presenting more of a broadface to light that is coming from a different angle (somewhat akin tothe manner in which plantation shutters may be utilized with a typicalhuman scale window). The MEMS/louvers configuration may be placed uponan optically transparent substrate, with the louvers substantiallyopaque. Ideally such a configuration would have a louver pitch fineenough to selectably occlude light on a pixel-by-pixel basis. In anotherembodiment, two or more layers or stacks of louvers may be combined toprovide yet further controls. In another embodiment, rather thanselectively blocking light, the louvers may be polarizers configured tochange the polarization state of light on a controllably variable basis.

As described above, another embodiment for selective occlusion maycomprise an array of sliding panels in a MEMS device such that thesliding panels may be controllably opened (i.e., by sliding in a planarfashion from a first position to a second position; or by rotating froma first orientation to a second orientation; or, for example, combinedrotational reorientation and displacement) to transmit light through asmall frame or aperture, and controllably closed to occlude the frame oraperture and prevent transmission. The array may be configured to openor occlude the various frames or apertures such that they maximallyattenuate the rays that are to be attenuated, and only minimallyattenuate the rays to be transmitted.

In an embodiment wherein a fixed number of sliding panels can eitheroccupy a first position occluding a first aperture and opening a secondaperture, or a second position occluding the second aperture and openingthe first aperture, there will always be the same amount of lighttransmitted overall (because 50% of the apertures are occluded, and theother 50% are open, with such a configuration), but the local positionchanges of the shutters or doors may create targeted moire or othereffects for darkfield perception with the dynamic positioning of thevarious sliding panels. In one embodiment, the sliding panels maycomprise sliding polarizers, and if placed in a stacked configurationwith other polarizing elements that are either static or dynamic, may beutilized to selectively attenuate.

Referring to FIG. 19D, another configuration providing an opportunityfor selective reflection, such as via a DMD style reflector array (534),is shown, such that a stacked set of two waveguides (536, 538) alongwith a pair of focus elements (540, 542) and a reflector (534; such as aDMD) may be used to capture a portion of incoming light with an entrancereflector (544). The reflected light may be totally internally reflecteddown the length of the first waveguide (536), into a focusing element(540) to bring the light into focus on a reflector (534) such as a DMDarray, after which the DMD may selectively attenuate and reflect aportion of the light back through a focusing lens (542; the lensconfigured to facilitate injection of the light back into the secondwaveguide) and into the second waveguide (538) for total internalreflection down to an exit reflector (546) configured to exit the lightout of the waveguide and toward the eye (58).

Such a configuration may have a relatively thin shape factor, and isdesigned to allow light from the real world (144) to be selectivelyattenuated. As waveguides work most cleanly with collimated light, sucha configuration may be well suited for virtual reality configurationswherein focal lengths are in the range of optical infinity. For closerfocal lengths, a lightfield display may be used as a layer on top of thesilhouette created by the aforementioned selective attenuation/darkfieldconfiguration to provide other cues to the eye of the user that light iscoming from another focal distance. An occlusion mask may be out offocus, even nondesirably so, and then in one embodiment, a lightfield ontop of the masking layer may be used to hide the fact that the darkfieldmay be at the wrong focal distance.

Referring to FIG. 19E, an embodiment is shown featuring two waveguides(552, 554) each having two angled reflectors (558, 544; 556, 546) forillustrative purposes shown at approximately 45 degrees; in actualconfigurations the angle may differ depending upon the reflectivesurface, reflective/refractive properties of the waveguides, etc.)directing a portion of light incoming from the real world down each sideof a first waveguide (or down two separate waveguides if the top layeris not monolithic) such that it hits a reflector (548, 550) at each end,such as a DMD which may be used for selective attenuation, after whichthe reflected light may be injected back into the second waveguide (orinto two separate waveguides if the bottom layer is not monolithic) andback toward two angled reflectors (again, they need not be at 45 degreesas shown) for exit out toward the eye (58).

Focusing lenses may also be placed between the reflectors at each endand the waveguides. In another embodiment the reflectors (548, 550) ateach end may comprise standard mirrors (such as alumized mirrors).Further, the reflectors may be wavelength selective reflectors, such asdichroic mirrors or film interference filters. Further, the reflectorsmay be diffractive elements configured to reflect incoming light.

FIG. 19F illustrates a configuration wherein four reflective surfaces ina pyramid type configuration are utilized to direct light through twowaveguides (560, 562), in which incoming light from the real world maybe divided up and reflected to four difference axes. The pyramid-shapedreflector (564) may have more than four facets, and may be residentwithin the substrate prism, as with the reflectors of the configurationof FIG. 19E. The configuration of FIG. 19F is an extension of that ofFIG. 19E.

Referring to FIG. 19G, a single waveguide (566) may be utilized tocapture light from the world (144) with one or more reflective surfaces(574, 576, 578, 580, 582), relay it (570) to a selective attenuator(568; such as a DMD array), and recouple it back into the same waveguideso that it propagates (572) and encounters one or more other reflectivesurfaces (584, 586, 588, 590, 592) that cause it to at least partiallyexit (594) the waveguide on a path toward the eye (58) of the user.Preferably the waveguide comprises selective reflectors such that onegroup (574, 576, 578, 580, 582) may be switched on to capture incominglight and direct it down to the selective attenuator, while separateanother group (584, 586, 588, 590, 592) may be switched on to exit lightreturning from the selective attenuator out toward the eye (58).

For simplicity the selective attenuator is shown oriented substantiallyperpendicularly to the waveguide; in other embodiments, various opticscomponents, such as refractive or reflective optics, may be utilized tohave the selective attenuator at a different and more compactorientation relative to the waveguide.

Referring to FIG. 19H, a variation on the configuration described inreference to FIG. 19D is illustrated. This configuration is somewhatanalogous to that discussed above in reference to FIG. 5B, wherein aswitchable array of reflectors may be embedded within each of a pair ofwaveguides (602, 604). Referring to FIG. 19H, a controller may beconfigured to turn the reflectors (598, 600) on and off in sequence,such that multiple reflectors may be operated on a frame sequentialbasis; then the DMD or other selective attenuator (594) may also besequentially driven in sync with the different mirrors being turned onand off.

Referring to FIG. 19I, a pair of wedge-shaped waveguides similar tothose described above (for example, in reference to FIGS. 15A-15C) areshown in side or sectional view to illustrate that the two long surfacesof each wedge-shaped waveguide (610, 612) are not co-planar. A “turningfilm” (606, 608; such as that available from 3M corporation under thetrade name, “TRAF”, which in essence comprises a microprism array), maybe utilized on one or more surfaces of the wedge-shaped waveguides toeither turn incoming rays at an angle so that they will be captured bytotal internal reflection, or to turn outgoing rays as they are exitingthe waveguide toward an eye or other target. Incoming rays are directeddown the first wedge and toward the selective attenuator (614) such as aDMD, LCD (such as a ferroelectric LCD), or an LCD stack to act as amask).

After the selective attenuator (614), reflected light is coupled backinto the second wedge-shaped waveguide which then relays the light bytotal internal reflection along the wedge. The properties of thewedge-shaped waveguide are intentionally such that each bounce of lightcauses an angle change; the point at which the angle has changed enoughto be the critical angle to escape total internal reflection becomes theexit point from the wedge-shaped waveguide. Typically the exit will beat an oblique angle, so another layer of turning film may be used to“turn” the exiting light toward a targeted object such as the eye (58).

Referring to FIG. 19J, several arcuate lenslet arrays (616, 620, 622)are positioned relative to an eye and configured such that a spatialattenuator array (618) is positioned at a focal/image plane so that itmay be in focus with the eye (58). The first (616) and second (620)arrays are configured such that in the aggregate, light passing from thereal world to the eye is essentially passed through a zero powertelescope. The embodiment of FIG. 19J shows a third array (622) oflenslets which may be utilized for improved optical compensation, butthe general case does not require such a third layer. As discussedabove, having telescopic lenses that are the diameter of the viewingoptic may create an undesirably large form factor (somewhat akin tohaving a bunch of small sets of binoculars in front of the eyes).

One way to optimize the overall geometry is to reduce the diameter ofthe lenses by splitting them out into smaller lenslets, as shown in FIG.19J (i.e., an array of lenses rather than one single large lens). Thelenslet arrays (616, 620, 622) are shown wrapped radially or arcuatelyaround the eye (58) to ensure that beams incoming to the pupil arealigned through the appropriate lenslets (else the system may sufferfrom optical problems such as dispersion, aliasing, and/or lack offocus). Thus all of the lenslets are oriented “toed in” and pointed atthe pupil of the eye (58), and the system facilitates avoidance ofscenarios wherein rays are propagated through unintended sets of lensesen route to the pupil.

Referring to FIGS. 19K-19N, various software approaches may be utilizedto assist in the presentation of darkfield in a virtual or augmentedreality displace scenario. Referring to FIG. 19K, a typical challengingscenario for augmented reality is depicted (632), with a textured carpet(624) and non-uniform background architectural features (626), both ofwhich are lightly-colored. The black box (628) depicted indicates theregion of the display in which one or more augmented reality featuresare to be presented to the user for three-dimensional perception, and inthe black box a robot creature (630) is being presented that may, forexample, be part of an augmented reality game in which the user isengaged. In the depicted example, the robot character (630) isdarkly-colored, which makes for a challenging presentation inthree-dimensional perception, particularly with the background selectedfor this example scenario.

As discussed briefly above, one of the main challenges for a presentingdarkfield augmented reality object is that the system generally cannotadd or paint in “darkness”; generally the display is configured to addlight. Thus, referring to FIG. 19L, without any specialized softwaretreatments to enhance darkfield perception, presentation of the robotcharacter in the augmented reality view results in a scene whereinportions of the robot character that are to be essentially flat black inpresentation are not visible, and portions of the robot character thatare to have some lighting (such as the lightly-pigmented cover of theshoulder gun of the robot character) are only barely visible (634)—theyappear almost like a light grayscale disruption to the otherwise normalbackground image.

Referring to FIG. 19M, using a software-based global attenuationtreatment (akin to digitally putting on a pair of sunglasses) providesenhanced visibility to the robot character because the brightness of thenearly black robot character is effective increased relative to the restof the space, which now appears more dark (640). Also shown in FIG. 19Mis a digitally-added light halo (636) which may be added to enhance anddistinguish the now-more-visible robot character shapes (638) from thebackground. With the halo treatment, even the portions of the robotcharacter that are to be presented as flat black become visible with thecontrast to the white halo, or “aura” presented around the robotcharacter.

Preferably the halo may be presented to the user with a perceived focaldistance that is behind the focal distance of the robot character inthree-dimensional space. In a configuration wherein single panelocclusion techniques such as those described above is being utilized topresent darkfield, the light halo may be presented with an intensitygradient to match the dark halo that may accompany the occlusion,minimizing the visibility of either darkfield effect. Further, the halomay be presented with blurring to the background behind the presentedhalo illumination for further distinguishing effect. A more subtle auraor halo effect may be created by matching, at least in part, the colorand/or brightness of a relatively light-colored background.

Referring to FIG. 19N, some or all of the black intonations of the robotcharacter may be changed to dark, cool blue colors to provide a furtherdistinguishing effect relative to the background, and relatively goodvisualization of the robot (642).

Wedge-shaped waveguides have been described above, such as in referenceto FIGS. 15A-15D and FIG. 19I. With a wedge-shaped waveguide, every timea ray bounces off of one of the non-coplanar surfaces, it gets an anglechange, which ultimately results in the ray exiting total internalreflection when its approach angle to one of the surfaces goes past thecritical angle. Turning films may be used to redirect exiting light sothat exiting beams leave with a trajectory that is more or lessperpendicular to the exit surface, depending upon the geometric andergonomic issues at play.

With a series or array of displays injecting image information into awedge-shaped waveguide, as shown in FIG. 15C, for example, thewedge-shaped waveguide may be configured to create a fine-pitched arrayof angle-biased rays emerging from the wedge. Somewhat similarly, it hasbeen discussed above that a lightfield display, or a variable wavefrontcreating waveguide, both may produce a multiplicity of beamlets or beamsto represent a single pixel in space such that wherever the eye ispositioned, the eye is hit by a plurality of different beamlets or beamsthat are unique to that particular eye position in front of the displaypanel.

As was further discussed above in the context of lightfield displays, aplurality of viewing zones may be created within a given pupil, and eachmay be used for a different focal distance, with the aggregate producinga perception similar to that of a variable wavefront creating waveguide,or similar to the actual optical physics of reality of the objectsviewed were real. Thus a wedge-shaped waveguide with multiple displaysmay be utilized to generate a lightfield. In an embodiment similar tothat of FIG. 15C with a linear array of displays injecting imageinformation, a fan of exiting rays is created for each pixel. Thisconcept may be extended in an embodiment wherein multiple linear arraysare stacked to all inject image information into the wedge-shapedwaveguide (in one variation, one array may inject at one angle relativeto the wedge-shaped waveguide face, while the second array may inject ata second angle relative to the wedge-shaped waveguide face), in whichcase exit beams fan out at two different axes from the wedge.

Thus such a configuration may be utilized to produce pluralities ofbeams spraying out at lots of different angles, and each beam may bedriven separately due to the fact that under such configuration, eachbeam is driven using a separate display. In another embodiment, one ormore arrays or displays may be configured to inject image informationinto wedge-shaped waveguide through sides or faces of the wedge-shapedwaveguide other than that shown in FIG. 15C, such as by using adiffractive optic to bend injected image information into total aninternal reflection configuration relative to the wedge-shapedwaveguide.

Various reflectors or reflecting surfaces may also be utilized inconcert with such a wedge-shaped waveguide embodiment to outcouple andmanage light from the wedge-shaped waveguide. In one embodiment, anentrance aperture to a wedge-shaped waveguide, or injection of imageinformation through a different face other than shown in FIG. 15C, maybe utilized to facilitate staggering (geometric and/or temporal) ofdifferent displays and arrays such that a Z-axis delta may also bedeveloped as a means for injecting three-dimensional information intothe wedge-shaped waveguide. For a greater than three-dimensions arrayconfiguration, various displays may be configured to enter awedge-shaped waveguide at multiple edges in multiple stacks withstaggering to get higher dimensional configurations.

Referring to FIG. 20A, a configuration similar to that depicted in FIG.8H is shown wherein a waveguide (646) has a diffractive optical element(648; or “DOE”, as noted above) sandwiched in the middle (alternatively,as described above, the diffractive optical element may reside on thefront or back face of the depicted waveguide). A ray may enter thewaveguide (646) from the projector or display (644). Once in thewaveguide (646), each time the ray intersects the DOE (648), part of itis exited out of the waveguide (646). As described above, the DOE may bedesigned such that the exit illuminance across the length of thewaveguide (646) is somewhat uniform (for example, the first such DOEintersection may be configured to exit about 10% of the light; then thesecond DOE intersection may be configured to exit about 10% of theremaining light so that 81% is passed on, and so on; in another embodieda DOE may be designed to have a variable diffraction efficiency, such aslinearly-decreasing diffraction efficiency, along its length to map outa more uniform exit illuminance across the length of the waveguide).

To further distribute remaining light that reaches an end (and in oneembodiment to allow for selection of a relatively low diffractionefficiency DOE which would be favorable from a view-to-the-worldtransparency perspective), a reflective element (650) at one or bothends may be included. Further, referring to the embodiment of FIG. 20B,additional distribution and preservation may be achieved by including anelongate reflector (652) across the length of the waveguide as shown(comprising, for example, a thin film dichroic coating that iswavelength-selective); preferably such reflector would be blocking lightthat accidentally is reflected upward (back toward the real world 144for exit in a way that it would not be utilized by the viewer). In someembodiments, such an elongate reflector may contribute to a “ghosting”effect perception by the user.

In one embodiment, this ghosting effect may be eliminated by having adual-waveguide (646, 654) circulating reflection configuration, such asthat shown in FIG. 20C, which is designed to keep the light movingaround until it has been exited toward the eye (58) in a preferablysubstantially equally distributed manner across the length of thewaveguide assembly. Referring to FIG. 20C, light may be injected with aprojector or display (644), and as it travels across the DOE (656) ofthe first waveguide (654), it ejects a preferably substantially uniformpattern of light out toward the eye (58); light that remains in thefirst waveguide is reflected by a first reflector assembly (660) intothe second waveguide (646). In one embodiment, the second waveguide(646) may be configured to not have a DOE, such that it merelytransports or recycles the remaining light back to the first waveguide,using the second reflector assembly.

In another embodiment (as shown in FIG. 20C) the second waveguide (646)may also have a DOE (648) configured to uniformly eject fractions oftravelling light to provide a second plane of focus forthree-dimensional perception. Unlike the configurations of FIGS. 20A and20B, the configuration of FIG. 20C is designed for light to travel thewaveguide in one direction, which avoids the aforementioned ghostingproblem that is related to passing light backwards through a waveguidewith a DOE. Referring to FIG. 20D, rather than having a mirror or boxstyle reflector assembly (660) at the ends of a waveguide for recyclingthe light, an array of smaller retroreflectors (662), or aretroreflective material, may be utilized.

Referring to FIG. 20E, an embodiment is shown that utilizes some of thelight recycling configurations of the embodiment of FIG. 20C to “snake”the light down through a waveguide (646) having a sandwiched DOE (648)after it has been injected with a display or projector (644) so that itcrosses the waveguide (646) many times back and forth before reachingthe bottom, at which point it may be recycled back up to the top levelfor further recycling. Such a configuration not only recycles the lightand facilitates use of relatively low diffraction efficiency DOEelements for exiting light toward the eye (58), but also distributes thelight, to provide for a large exit pupil configuration akin to thatdescribed in reference to FIG. 8K.

Referring to FIG. 20F, an illustrative configuration similar to that ofFIG. 5A is shown, with incoming light injected along a conventionalprism or beamsplitter substrate (104) to a reflector (102) without totalinternal reflection (i.e., without the prism being considered awaveguide) because the input projection (106), scanning or otherwise, iskept within the bounds of the prism—which means that the geometry ofsuch prism becomes a significant constraint. In another embodiment, awaveguide may be utilized in place of the simple prism of FIG. 20F,which facilitates the use of total internal reflection to provide moregeometric flexibility.

Other configurations describe above are configured to profit from theinclusion of waveguides for similar manipulations and light. Forexample, referring back to FIG. 7A, the general concept illustratedtherein is that a collimated image injected into a waveguide may berefocused before transfer out toward an eye, in a configuration alsodesigned to facilitate viewing light from the real world. In place ofthe refractive lens shown in FIG. 7A, a diffractive optical element maybe used as a variable focus element.

Referring back to FIG. 7B, another waveguide configuration isillustrated in the context of having multiple layers stacked upon eachother with controllable access toggling between a smaller path (totalinternal reflection through a waveguide) and a larger path (totalinternal reflection through a hybrid waveguide comprising the originalwaveguide and a liquid crystal isolated region with the liquid crystalswitched to a mode wherein the refractive indices are substantiallymatched between the main waveguide and the auxiliary waveguide), so thatthe controller can tune on a frame-by-frame basis which path is beingtaken. High-speed switching electro-active materials, such as lithiumniobate, facilitate path changes with such a configuration at gigahertzrates, which allows one to change the path of light on a pixel-by-pixelbasis.

Referring back to FIG. 8A, a stack of waveguides paired with weak lensesis illustrated to demonstrate a multifocal configuration wherein thelens and waveguide elements may be static. Each pair of waveguide andlens may be functionally replaced with waveguide having an embedded DOEelement (which may be static, in a closer analogy to the configurationof FIG. 8A, or dynamic), such as that described in reference to FIG. 8I.

Referring to FIG. 20G, if a transparent prism or block (104; i.e., not awaveguide) is utilized to hold a mirror or reflector (102) in aperiscope type of configuration to receive light from other components,such as a lens (662) and projector or display (644), the field of viewis limited by the size of that reflector (102; the bigger the reflector,the wider the field of view). Thus to have a larger field of view withsuch configuration, a thicker substrate may be needed to hold a largerreflector; otherwise, the functionality of an aggregated plurality ofreflectors may be utilized to increase the functional field of view, asdescribed in reference to FIGS. 8O, 8P, and 8Q. Referring to FIG. 20H, astack (664) of planar waveguides (666), each fed with a display orprojector (644; or in another embodiment a multiplexing of a singledisplay) and having an exit reflector (668), may be utilized toaggregate toward the function of a larger single reflector. The exitreflectors may be at the same angle in some cases, or not the same anglein other cases, depending upon the positioning of the eye (58) relativeto the assembly.

FIG. 20I illustrates a related configuration, wherein the reflectors(680, 682, 684, 686, 688) in each of the planar waveguides (670, 672,674, 676, 678) have been offset from each other, and wherein each takesin light from a projector or display (644) which may be sent through alens (690) to ultimately contribute exiting light to the pupil (45) ofthe eye (58) by virtue of the reflectors (680, 682, 684, 686, 688) ineach of the planar waveguides (670, 672, 674, 676, 678). If one cancreate a total range of all of the angles that would be expected to beseen in the scene (i.e., preferably without blind spots in the key fieldof view), then a useful field of view has been achieved. As describedabove, the eye (58) functions based at least on what angle light raysenter the eye, and this can be simulated. The rays need not pass throughthe exact same point in space at the pupil—rather the light rays justneed to get through the pupil and be sensed by the retina. FIG. 20Killustrates a variation wherein the shaded portion of the opticalassembly may be utilized as a compensating lens to functionally passlight from the real world (144) through the assembly as though it hasbeen passed through a zero power telescope.

Referring to FIG. 20J, each of the aforementioned rays may also be arelative wide beam that is being reflected through the pertinentwaveguide (670, 672) by total internal reflection. The reflector (680,682) facet size will determine what the exiting beam width can be.

Referring to FIG. 20L, a further discretization of the reflector isshown, wherein a plurality of small straight angular reflectors may forma roughly parabolic reflecting surface (694) in the aggregate through awaveguide or stack thereof (696). Light coming in from the displays(644; or single MUXed display, for example), such as through a lens(690), is all directed toward the same shared focal point at the pupil(45) of the eye (58).

Referring back to FIG. 13M, a linear array of displays (378) injectslight into a shared waveguide (376). In another embodiment a singledisplay may be multiplexed to a series of entry lenses to providesimilar functionality as the embodiment of FIG. 13M, with the entrylenses creating parallel paths of rays running through the waveguide.

In a conventional waveguide approach wherein total internal reflectionis relied upon for light propagation, the field of view is restrictedbecause there is only a certain angular range of rays propagatingthrough the waveguide (others may escape out). In one embodiment, if ared/green/blue (or “RGB”) laserline reflector is placed at one or bothends of the planar surfaces, akin to a thin film interference filterthat is highly reflective for only certain wavelengths and poorlyreflective for other wavelengths, than one can functionally increase therange of angles of light propagation. Windows (without the coating) maybe provided for allowing light to exit in predetermined locations.Further, the coating may be selected to have a directional selectivity(somewhat like reflective elements that are only highly reflective forcertain angles of incidence). Such a coating may be most relevant forthe larger planes/sides of a waveguide.

Referring back to FIG. 13E, a variation on a scanning fiber display wasdiscussed, which may be deemed a scanning thin waveguide configuration,such that a plurality of very thin planar waveguides (358) may beoscillated or vibrated such that if a variety of injected beams iscoming through with total internal reflection, the configurationfunctionally would provide a linear array of beams escaping out of theedges of the vibrating elements (358). The depicted configuration hasapproximately five externally-projecting planar waveguide portions (358)in a host medium or substrate (356) that is transparent, but whichpreferably has a different refractive index so that the light will stayin total internal reflection within each of the substrate-bound smallerwaveguides that ultimately feed (in the depicted embodiment there is a90 degree turn in each path at which point a planar, curved, or otherreflector may be utilized to bounce the light outward) theexternally-projecting planar waveguide portions (358).

The externally-projecting planar waveguide portions (358) may bevibrated individually, or as a group along with oscillatory motion ofthe substrate (356). Such scanning motion may provide horizontalscanning, and for vertical scanning, the input (360) aspect of theassembly (i.e., such as one or more scanning fiber displays scanning inthe vertical axis) may be utilized. Thus a variation of the scanningfiber display is presented.

Referring back to FIG. 13H, a waveguide (370) may be utilized to createa lightfield. With waveguides working best with collimated beams thatmay be associated with optical infinity from a perception perspective,all beams staying in focus may cause perception discomfort (i.e., theeye will not make a discernible difference in dioptric blur as afunction of accommodation; in other words, the narrow diameter, such as0.5 mm or less, collimated beamlets may open loop the eye'saccommodation/vergence system, causing discomfort).

In one embodiment, a single beam may be fed in with a number of conebeamlets coming out, but if the introduction vector of the entering beamis changed (i.e., laterally shift the beam injection location for theprojector/display relative to the waveguide), one may control where thebeam exits from the waveguide as it is directed toward the eye. Thus onemay use a waveguide to create a lightfield by creating a bunch of narrowdiameter collimated beams, and such a configuration is not reliant upona true variation in a light wavefront to be associated with the desiredperception at the eye.

If a set of angularly and laterally diverse beamlets is injected into awaveguide (for example, by using a multicore fiber and driving each coreseparately; another configuration may utilize a plurality of fiberscanners coming from different angles; another configuration may utilizea high-resolution panel display with a lenslet array on top of it), anumber of exiting beamlets can be created at different exit angles andexit locations. Since the waveguide may scramble the lightfield, thedecoding is preferably predetermined.

Referring to FIGS. 20M and 20N, a waveguide (646) assembly (696) isshown that comprises stacked waveguide components in the vertical orhorizontal axis. Rather than having one monolithic planar waveguide, thenotion with these embodiments is to stack a plurality of smallerwaveguides (646) immediately adjacent each other such that lightintroduced into one waveguide, in addition to propagating down (i.e.,propagating along a Z axis with total internal reflection in +X,−X) suchwaveguide by total internal reflection, also totally internally reflectsin the perpendicular axis (+y, −Y) as well, such that it is not spillinginto other areas. In other words, if total internal reflection is fromleft to right and back during Z axis propagation, the configuration willbe set up to totally internally reflect any light that hits the top orbottom sides as well; each layer may be driven separately withoutinterference from other layers. Each waveguide may have a DOE (648)embedded and configured to eject out light with a predetermineddistribution along the length of the waveguide, as described above, witha predetermined focal length configuration (shown in FIG. 20M as rangingfrom 0.5 meters to optical infinity).

In another variation, a very dense stack of waveguides with embeddedDOEs may be produced such that it spans the size of the anatomical pupilof the eye (i.e., such that multiple layers 698 of the compositewaveguide are required to cross the exit pupil, as illustrated in FIG.20N). With such a configuration, one may feed a collimated image for onewavelength, and then the portion located the next millimeter downproducing a diverging wavefront that represents an object coming from afocal distance of, say, 15 meters away, and so on, with the notion beingthat an exit pupil is coming from a number of different waveguides as aresult of the DOEs and total internal reflection through the waveguidesand across the DOEs. Thus rather than creating one uniform exit pupil,such a configuration creates a plurality of stripes that, in theaggregate, facilitate the perception of different focal depths with theeye/brain.

Such a concept may be extended to configurations comprising a waveguidewith a switchable/controllable embedded DOE (i.e. that is switchable todifferent focal distances), such as those described in relation to FIGS.8B-8N, which allows more efficient light trapping in the axis acrosseach waveguide. Multiple displays may be coupled into each of thelayers, and each waveguide with DOE would emit rays along its ownlength. In another embodiment, rather than relying on total internalreflection, a laserline reflector may be used to increase angular range.In between layers of the composite waveguide, a completely reflectivemetallized coating may be utilized, such as aluminum, to ensure totalreflection, or alternatively dichroic style or narrow band reflectorsmay be utilized.

Referring to FIG. 20O, the whole composite waveguide assembly (696)maybe be curved concavely toward the eye (58) such that each of theindividual waveguides is directed toward the pupil. In other words, theconfiguration may be designed to more efficiently direct the lighttoward the location where the pupil is likely to be present. Such aconfiguration also may be utilized to increase the field of view.

As was discussed above in relation to FIGS. 8L, 8M, and 8N, a changeablediffraction configuration allows for scanning in one axis, somewhat akinto a scanning light display. FIG. 21A illustrates a waveguide (698)having an embedded (i.e., sandwiched within) DOE (700) with a lineargrating term that may be changed to alter the exit angle of exitinglight (702) from the waveguide, as shown. A high-frequency switching DOEmaterial such as lithium niobate may be utilized. In one embodiment,such a scanning configuration may be used as the sole mechanism forscanning a beam in one axis; in another embodiment, the scanningconfiguration may be combined with other scanning axes, and may be usedto create a larger field of view (i.e., if a normal field of view is 40degrees, and by changing the linear diffraction pitch one can steer overanother 40 degrees, the effective usable field of view for the system is80 degrees).

Referring to FIG. 21B, in a conventional configuration, a waveguide(708) may be placed perpendicular to a panel display (704), such as anLCD or OLED panel, such that beams may be injected from the waveguide(708), through a lens (706), and into the panel (704) in a scanningconfiguration to provide a viewable display for television or otherpurposes. Thus the waveguide may be utilized in such configuration as ascanning image source, in contrast to the configurations described inreference to FIG. 21A, wherein a single beam of light may be manipulatedby a scanning fiber or other element to sweep through different angularlocations, and in addition, another direction may be scanned using thehigh-frequency diffractive optical element.

In another embodiment, a uniaxial scanning fiber display (say scanningthe fast line scan, as the scanning fiber is relatively high frequency)may be used to inject the fast line scan into the waveguide, and thenthe relatively slow DOE switching (i.e., in the range of 100 Hz) may beused to scan lines in the other axis to form an image.

In another embodiment, a DOE with a grating of fixed pitch may becombined with an adjacent layer of electro-active material having adynamic refractive index (such as liquid crystal), so that light may beredirected into the grating at different angles. This is an applicationof the basic multipath configuration described above in reference toFIG. 7B, in which an electro-active layer comprising an electro-activematerial such as liquid crystal or lithium niobate may change itsrefractive index such that it changes the angle at which a ray emergesfrom the waveguide. A linear diffraction grating may be added to theconfiguration of FIG. 7B (in one embodiment, sandwiched within the glassor other material comprising the larger lower waveguide) such that thediffraction grating may remain at a fixed pitch, but the light is biasedbefore it hits the grating.

FIG. 21C shows another embodiment featuring two wedge-like waveguideelements (710, 712), wherein one or more of them may be electro-activeso that the related refractive index may be changed. The elements may beconfigured such that when the wedges have matching refractive indices,the light totally internally reflects through the pair (which in theaggregate performs akin to a planar waveguide with both wedges matching)while the wedge interfaces have no effect. Then if one of the refractiveindices is changed to create a mismatch, a beam deflection at the wedgeinterface (714) is caused, and there is total internal reflection fromthat surface back into the associated wedge. Then a controllable DOE(716) with a linear grating may be coupled along one of the long edgesof the wedge to allow light to exit out and reach the eye at a desirableexit angle.

In another embodiment, a DOE such as a Bragg grating, may be configuredto change pitch versus time, such as by a mechanical stretching of thegrating (for example, if the grating resides on or comprises an elasticmaterial), a moire beat pattern between two gratings on two differentplanes (the gratings may be the same or different pitches), Z-axismotion (i.e., closer to the eye, or farther away from the eye) of thegrating, which functionally is similar in effect to stretching of thegrating, or electro-active gratings that may be switched on or off, suchas one created using a polymer dispersed liquid crystal approach whereinliquid crystal droplets may be controllably activated to change therefractive index to become an active grating, versus turning the voltageoff and allowing a switch back to a refractive index that matches thatof the host medium.

In another embodiment, a time-varying grating may be utilized for fieldof view expansion by creating a tiled display configuration. Further, atime-varying grating may be utilized to address chromatic aberration(failure to focus all colors/wavelengths at the same focal point). Oneproperty of diffraction gratings is that they will deflect a beam as afunction of its angle of incidence and wavelength (i.e., a DOE willdeflect different wavelengths by different angles: somewhat akin to themanner in which a simple prism will divide out a beam into itswavelength components).

One may use time-varying grating control to compensate for chromaticaberration in addition to field of view expansion. Thus, for example, ina waveguide with embedded DOE type of configuration as described above,the DOE may be configured to drive the red wavelength to a slightlydifferent place than the green and blue to address unwanted chromaticaberration. The DOE may be time-varied by having a stack of elementsthat switch on and off (i.e. to get red, green, and blue to bediffracted outbound similarly).

In another embodiment, a time-varying grating may be utilized for exitpupil expansion. For example, referring to FIG. 21D, it is possible thata waveguide (718) with embedded DOE (720) may be positioned relative toa target pupil such that none of the beams exiting in a baseline modeactually enter the target pupil (45)—such that the pertinent pixel wouldbe missed by the user. A time-varying configuration may be utilized tofill in the gaps in the outbound exit pattern by shifting the exitpattern laterally (shown in dashed/dotted lines) to effectively scaneach of the 5 exiting beams to better ensure that one of them hits thepupil of the eye. In other words, the functional exit pupil of thedisplay system is expanded.

In another embodiment, a time-varying grating may be utilized with awaveguide for one, two, or three axis light scanning. In a manner akinto that described in reference to FIG. 21A, one may use a term in agrating that is scanning a beam in the vertical axis, as well as agrating that is scanning in the horizontal axis. Further, if radialelements of a grating are incorporated, as is discussed above inrelation to FIGS. 8B-8N, one may have scanning of the beam in the Z axis(i.e., toward/away from the eye), all of which may be time sequentialscanning.

Notwithstanding the discussions herein regarding specialized treatmentsand uses of DOEs generally in connection with waveguides, many of theseuses of DOE are usable whether or not the DOE is embedded in awaveguide. For example, the output of a waveguide may be separatelymanipulated using a DOE; or a beam may be manipulated by a DOE before itis injected into a waveguide; further, one or more DOEs, such as atime-varying DOE, may be utilized as an input for freeform opticsconfigurations, as discussed below.

As discussed above in reference to FIGS. 8B-8N, an element of a DOE mayhave a circularly-symmetric term, which may be summed with a linear termto create a controlled exit pattern (i.e., as described above, the sameDOE that outcouples light may also focus it). In another embodiment, thecircular term of the DOE diffraction grating may be varied such that thefocus of the beams representing those pertinent pixels is modulated. Inaddition, one configuration may have a second/separate circular DOE,obviating the need to have a linear term in the DOE.

Referring to FIG. 21E, one may have a waveguide (722) outputtingcollimated light with no DOE element embedded, and a second waveguidethat has a circularly-symmetric DOE that can be switched betweenmultiple configurations—in one embodiment by having a stack (724) ofsuch DOE elements (FIG. 21F shows another configuration wherein afunctional stack 728 of DOE elements may comprise a stack of polymerdispersed liquid crystal elements 726, as described above, whereinwithout a voltage applied, a host medium refraction index matches thatof a dispersed molecules of liquid crystal; in another embodiment,molecules of lithium niobate may be dispersed for faster response times;with voltage applied, such as through transparent indium tin oxidelayers on either side of the host medium, the dispersed molecules changeindex of refraction and functionally form a diffraction pattern withinthe host medium) that can be switched on/off.

In another embodiment, a circular DOE may be layered in front of awaveguide for focus modulation. Referring to FIG. 21G, the waveguide(722) is outputting collimated light, which will be perceived asassociated with a focal depth of optical infinity unless otherwisemodified. The collimated light from the waveguide may be input into adiffractive optical element (730) which may be used for dynamic focusmodulation (i.e., one may switch on and off different circular DOEpatterns to impart various different focuses to the exiting light). In arelated embodiment, a static DOE may be used to focus collimated lightexiting from a waveguide to a single depth of focus that may be usefulfor a particular user application.

In another embodiment, multiple stacked circular DOEs may be used foradditive power and many focus levels—from a relatively small number ofswitchable DOE layers. In other words, three different DOE layers may beswitched on in various combinations relative to each other; the opticalpowers of the DOEs that are switched on may be added. In one embodimentwherein a range of up to 4 diopters is desired, for example, a first DOEmay be configured to provide half of the total diopter range desired (inthis example, 2 diopters of change in focus); a second DOE may beconfigured to induce a 1 diopter change in focus; then a third DOE maybe configured to induce a ½ diopter change in focus. These three DOEsmay be mixed and matched to provide ½, 1, 1.5, 2, 2.5, 3, and 3.5diopters of change in focus. Thus a super large number of DOEs would notbe required to get a relatively broad range of control.

In one embodiment, a matrix of switchable DOE elements may be utilizedfor scanning, field of view expansion, and/or exit pupil expansion.Generally in the above discussions of DOEs, it has been assume that atypical DOE is either all on or all off. In one variation, a DOE (732)may be subdivided into a plurality of functional subsections (such asthe one labeled as element 734 in FIG. 21H), each of which preferably isuniquely controllable to be on or off (for example, referring to FIG.21H, each subsection may be operated by its own set of indium tin oxide,or other control lead material, voltage application leads 736 back to acentral controller). Given this level of control over a DOE paradigm,additional configurations are facilitated.

Referring to FIG. 21I, a waveguide (738) with embedded DOE (740) isviewed from the top down, with the user's eye positioned in front of thewaveguide. A given pixel may be represented as a beam coming into thewaveguide and totally internally reflecting along until it may be exitedby a diffraction pattern to come out of the waveguide as a set of beams.Depending upon the diffraction configuration, the beams may come outparallel/collimated (as shown in FIG. 21I for convenience), or in adiverging fan configuration if representing a focal distance closer thanoptical infinity.

The depicted set of parallel exiting beams may represent, for example,the farthest left pixel of what the user is seeing in the real world asviewed through the waveguide, and light off to the rightmost extremewill be a different group of parallel exiting beams. Indeed, withmodular control of the DOE subsections as described above, one may spendmore computing resource or time creating and manipulating the smallsubset of beams that is likely to be actively addressing the user'spupil (i.e., because the other beams never reach the user's eye and areeffectively wasted). Thus, referring to FIG. 21J, a waveguide (738)configuration is shown wherein only the two subsections (740, 742) ofthe DOE (744) are deemed to be likely to address the user's pupil (45)are activated. Preferably one subsection may be configured to directlight in one direction simultaneously as another subsection is directinglight in a different direction.

FIG. 21K shows an orthogonal view of two independently controlledsubsections (734, 746) of a DOE (732). Referring to the top view of FIG.21L, such independent control may be used for scanning or focusinglight. In the configuration depicted in FIG. 21K, an assembly (748) ofthree independently controlled DOE/waveguide subsections (750, 752, 754)may be used to scan, increase the field of view, and/or increase theexit pupil region. Such functionality may arise from a single waveguidewith such independently controllable DOE subsections, or a verticalstack of these for additional complexity.

In one embodiment, if a circular DOE may be controllably stretchedradially-symmetrically, the diffraction pitch may be modulated, and theDOE may be utilized as a tunable lens with an analog type of control. Inanother embodiment, a single axis of stretch (for example, to adjust anangle of a linear DOE term) may be utilized for DOE control. Further, inanother embodiment a membrane, akin to a drum head, may be vibrated,with oscillatory motion in the Z-axis (i.e., toward/away from the eye)providing Z-axis control and focus change over time.

Referring to FIG. 21M, a stack of several DOEs (756) is shown receivingcollimated light from a waveguide (722) and refocusing it based upon theadditive powers of the activated DOEs. Linear and/or radial terms ofDOEs may be modulated over time, such as on a frame sequential basis, toproduce a variety of treatments (such as tiled display configurations orexpanded field of view) for the light coming from the waveguide andexiting, preferably toward the user's eye. In configurations wherein theDOE or DOEs are embedded within the waveguide, a low diffractionefficiency is desired to maximize transparency for light passed from thereal world; in configurations wherein the DOE or DOEs are not embedded,a high diffraction efficiency may be desired, as described above. In oneembodiment, both linear and radial DOE terms may be combined outside ofthe waveguide, in which case high diffraction efficiency would bedesired.

Referring to FIG. 21N, a segmented or parabolic reflector, such as thosediscussed above in FIG. 8Q, is shown. Rather than executing a segmentedreflector by combining a plurality of smaller reflectors, in oneembodiment the same functionality may result from a single waveguidewith a DOE having different phase profiles for each section of it, suchthat it is controllable by subsection. In other words, while the entiresegmented reflector functionality may be turned on or off together,generally the DOE may be configured to direct light toward the sameregion in space (i.e., the pupil of the user).

Referring to FIGS. 22A-22Z, optical configurations known as “freeformoptics” may be utilized certain of the aforementioned challenges. Theterm “freeform” generally is used in reference to arbitrarily curvedsurfaces that may be utilized in situations wherein a spherical,parabolic, or cylindrical lens does not meet a design complexity such asa geometric constraint. For example, referring to FIG. 22A, one of thecommon challenges with display (762) configurations when a user islooking through a mirror (and also sometimes a lens 760) is that thefield of view is limited by the area subtended by the final lens (760)of the system.

Referring to FIG. 22B, in more simple terms, if one has a display (762),which may include some lens elements, there is a straightforwardgeometric relationship such that the field of view cannot be larger thanthe angle subtended by the display (762). Referring to FIG. 22C, thischallenge is exacerbated if the user is trying to have an augmentedreality experience wherein light from the real world is also be topassed through the optical system, because in such case, there often isa reflector (764) that leads to a lens (760); by interposing areflector, the overall path length to get to the lens from the eye isincreased, which tightens the angle and reduces the field of view.

Given this, if one wants to increase the field of view, he must increasethe size of the lens, but that might mean pushing a physical lens towardthe forehead of the user from an ergonomic perspective. Further, thereflector may not catch all of the light from the larger lens. Thus,there is a practical limitation imposed by human head geometry, and itgenerally is a challenge to get more than a 40-degree field of viewusing conventional see-through displays and lenses.

With freeform lenses, rather than having a standard planar reflector asdescribed above, one has a combined reflector and lens with power (i.e.,a curved reflector 766), which means that the curved lens geometrydetermines the field of view. Referring to FIG. 22D, without thecircuitous path length of a conventional paradigm as described above inreference to FIG. 22C, it is possible for a freeform arrangement torealize a significantly larger field of view for a given set of opticalrequirements.

Referring to FIG. 22E, a typical freeform optic has three activesurfaces. Referring to FIG. 22E, in one typical freeform optic (770)configuration, light may be directed toward the freeform optic from animage plane, such as a flat panel display (768), into the first activesurface (772), which typically is a primarily transmissive freeformsurface that refracts transmitted light and imparts a focal change (suchas an added stigmatism, because the final bounce from the third surfacewill add a matching/opposite stigmatism and these are desirablycanceled). The incoming light may be directed from the first surface toa second surface (774), wherein it may strike with an angle shallowenough to cause the light to be reflected under total internalreflection toward the third surface (776).

The third surface may comprise a half-silvered, arbitrarily-curvedsurface configured to bounce the light out through the second surfacetoward the eye, as shown in FIG. 22E. Thus in the depicted typicalfreeform configuration, the light enters through the first surface,bounces from the second surface, bounces from the third surface, and isdirected out of the second surface. Due to the optimization of thesecond surface to have the requisite reflective properties on the firstpass, as well as refractive properties on the second pass as the lightis exited toward the eye, a variety of curved surfaces with higher-ordershapes than a simple sphere or parabola are formed into the freeformoptic.

Referring to FIG. 22F, a compensating lens (780) may be added to thefreeform optic (770) such that the total thickness of the optic assemblyis substantially uniform in thickness, and preferably withoutmagnification, to light incoming from the real world (144) in anaugmented reality configuration.

Referring to FIG. 22G, a freeform optic (770) may be combined with awaveguide (778) configured to facilitate total internal reflection ofcaptured light within certain constraints. For example, as shown in FIG.22G, light may be directed into the freeform/waveguide assembly from animage plane, such as a flat panel display, and totally internallyreflected within the waveguide until it hits the curved freeform surfaceand escapes toward the eye of the user. Thus the light bounces severaltimes in total internal reflection until it reaches the freeform wedgeportion.

One of the main objectives with such an assembly is to try to lengthenthe optic assembly while retaining as uniform a thickness as possible(to facilitate transport by total internal reflection, and also viewingof the world through the assembly without further compensation) for alarger field of view. FIG. 22H depicts a configuration similar to thatof FIG. 22G, with the exception that the configuration of FIG. 22H alsofeatures a compensating lens portion to further extend the thicknessuniformity and assist with viewing the world through the assemblywithout further compensation.

Referring to FIG. 22I, in another embodiment, a freeform optic (782) isshown with a small flat surface, or fourth face (784), at the lower leftcorner that is configured to facilitate injection of image informationat a different location than is typically used with freeform optics. Theinput device (786) may comprise, for example, a scanning fiber display,which may be designed to have a very small output geometry. The fourthface may comprise various geometries itself and have its own refractivepower, such as by use planar or freeform surface geometries.

Referring to FIG. 22J, in practice, such a configuration may alsofeature a reflective coating (788) along the first surface such that itdirects light back to the second surface, which then bounces the lightto the third surface, which directs the light out across the secondsurface and to the eye (58). The addition of the fourth small surfacefor injection of the image information facilitates a more compactconfiguration. In an embodiment wherein a classical freeform inputconfiguration and a scanning fiber display (790) are utilized, somelenses (792, 794) may be required in order to appropriately form animage plane (796) using the output from the scanning fiber display;these hardware components add extra bulk that may not be desired.

Referring to FIG. 22K, an embodiment is shown wherein light from ascanning fiber display (790) is passed through an input optics assembly(792, 794) to an image plane (796), and then directed across the firstsurface of the freeform optic (770) to a total internal reflectionbounce off of the second surface, then another total internal reflectionbounce from the third surface results in the light exiting across thesecond surface and being directed toward the eye (58).

An all-total-internal-reflection freeform waveguide may be created suchthat there are no reflective coatings (i.e., such thattotal-internal-reflection is being relied upon for propagation of lightuntil a critical angle of incidence with a surface is met, at whichpoint the light exits in a manner akin to the wedge-shaped opticsdescribed above). In other words, rather than having two planarsurfaces, one may have a surface comprising one or more sub-surfacesfrom a set of conical curves, such as parabolas, spheres, ellipses,etc.).

Such a configuration still may produce a shallow-enough angles for totalinternal reflection within the optic; thus an approach that is somewhata hybrid between a conventional freeform optic and a wedge-shapedwaveguide is presented. One motivation to have such a configuration isto get away from the use of reflective coatings, which do help productreflection, but also are known to prevent transmission of a relativelylarge portion (such as 50%) of the light transmitting through from thereal world (144). Further, such coatings also may block an equivalentamount of the light coming into the freeform optic from the inputdevice. Thus there are reasons to develop designs that do not havereflective coatings.

As described above, one of the surfaces of a conventional freeform opticmay comprise a half-silvered reflective surface. Generally such areflective surface will be of “neutral density”, meaning that it willgenerally reflect all wavelengths similarly. In another embodiment, suchas one wherein a scanning fiber display is utilized as an input, theconventional reflector paradigm may be replaced with a narrow bandreflector that is wavelength sensitive, such as a thin film laserlinereflector. Thus in one embodiment, a configuration may reflectparticular red/green/blue wavelength ranges and remain passive to otherwavelengths, which generally will increase transparency of the optic andtherefore be preferred for augmented reality configurations whereintransmission of image information from the real world (144) across theoptic also is valued.

Referring to FIG. 22L, an embodiment is depicted wherein multiplefreeform optics (770) may be stacked in the Z axis (i.e., along an axissubstantially aligned with the optical axis of the eye). In onevariation, each of the three depicted freeform optics may have awavelength-selective coating (for example, one highly selective forblue, the next for green, the next for red) so that images may beinjected into each to have blue reflected from one surface, green fromanother, and red from a third surface. Such a configuration may beutilized, for example, to address chromatic aberration issues, to createa lightfield, or to increase the functional exit pupil size.

Referring to FIG. 22M, an embodiment is shown wherein a single freeformoptic (798) has multiple reflective surfaces (800, 802, 804), each ofwhich may be wavelength or polarization selective so that theirreflective properties may be individually controlled.

Referring to FIG. 22N, in one embodiment, multiple microdisplays, suchas scanning light displays, (786) may be injected into a single freeformoptic to tile images (thereby providing an increased field of view),increase the functional pupil size, or address challenges such aschromatic aberration (i.e., by reflecting one wavelength per display).Each of the depicted displays would inject light that would take adifferent path through the freeform optic due to the differentpositioning of the displays relative to the freeform optic, which wouldprovide a larger functional exit pupil output.

In one embodiment, a packet or bundle of scanning fiber displays may beutilized as an input to overcome one of the challenges in operativelycoupling a scanning fiber display to a freeform optic. One suchchallenge with a scanning fiber display configuration is that the outputof an individual fiber is emitted with a certain numerical aperture, or“NA”, which is like the projectional angle of light from the fiber;ultimately this angle determines the diameter of the beam that passesthrough various optics, and ultimately determines the exit functionalexit pupil size; thus in order to maximize exit pupil size with afreeform optic configuration, one may either increase the NA of thefiber using optimized refractive relationships, such as between core andcladding, or one may place a lens (i.e., a refractive lens, such as agradient refractive index lens, or “GRIN” lens) at the end of the fiberor build one into the end of the fiber as described above, or create anarray of fibers that is feeding into the freeform optic, in which caseall of those NAs in the bundle remain small, and at the exit pupil anarray of small exit pupils is produced that in the aggregate forms thefunctional equivalent of a large exit pupil.

Alternatively, in another embodiment a more sparse array (i.e., notbundled tightly as a packet) of scanning fiber displays or otherdisplays may be utilized to functionally increase the field of view ofthe virtual image through the freeform optic. Referring to FIG. 22O, inanother embodiment, a plurality of displays or displays (786) may beinjected through the top of a freeform optic (770), as well as anotherplurality (786) through the lower corner; the display arrays may be twoor three dimensional arrays. Referring to FIG. 22P, in another relatedembodiment, image information also may be injected in from the side(806) of the freeform optic (770) as well.

In an embodiment wherein a plurality of smaller exit pupils is to beaggregated into a functionally larger exit pupil, one may elect to haveeach of the scanning fibers monochromatic, such that within a givenbundle or plurality of projectors or displays, one may have a subgroupof solely red fibers, a subgroup of solely blue fibers, and a subgroupof solely green fibers. Such a configuration facilitates more efficiencyin output coupling for bringing light into the optical fibers; forinstance, there would be no need in such an embodiment to superimposered, green, and blue into the same band.

Referring to FIGS. 22Q-22V, various freeform optic tiling configurationsare depicted. Referring to FIG. 22Q, an embodiment is depicted whereintwo freeform optics are tiled side-by-side and a microdisplay, such as ascanning light display, (786) on each side is configured to inject imageinformation from each side, such that one freeform optic wedgerepresents each half of the field of view.

Referring to FIG. 22R, a compensator lens (808) may be included tofacilitate views of the real world through the optics assembly. FIG. 22Sillustrates a configuration wherein freeform optics wedges are tiledside by side to increase the functional field of view while keeping thethickness of such optical assembly relatively uniform.

Referring to FIG. 22T, a star-shaped assembly comprises a plurality offreeform optics wedges (also shown with a plurality of displays forinputting image information) in a configuration that may provide alarger field of view expansion while also maintaining a relatively thinoverall optics assembly thickness.

With a tiled freeform optics assembly, the optics elements may beaggregated to produce a larger field of view; the tiling configurationsdescribed above have addressed this notion. For example, in aconfiguration wherein two freeform waveguides are aimed at the eye suchas that depicted in FIG. 22R, there are several ways to increase thefield of view. One option is to “toe in” the freeform waveguides suchthat their outputs share, or are superimposed in, the space of the pupil(for example, the user may see the left half of the visual field throughthe left freeform waveguide, and the right half of the visual fieldthrough the right freeform waveguide).

With such a configuration, the field of view has been increased with thetiled freeform waveguides, but the exit pupil has not grown in size.Alternatively, the freeform waveguides may be oriented such that they donot toe in as much—so they create exit pupils that are side-by-side atthe eye's anatomical pupil. In one example, the anatomical pupil may be8 mm wide, and each of the side-by-side exit pupils may be 8 mm, suchthat the functional exit pupil is expanded by about two times. Thus sucha configuration provides an enlarged exit pupil, but if the eye is movedaround in the “eyebox” defined by that exit pupil, that eye may loseparts of the visual field (i.e., lose either a portion of the left orright incoming light because of the side-by-side nature of suchconfiguration).

In one embodiment using such an approach for tiling freeform optics,especially in the Z-axis relative to the eye of the user, redwavelengths may be driven through one freeform optic, green throughanother, and blue through another, such red/green/blue chromaticaberration may be addressed. Multiple freeform optics also may beprovided to such a configuration that are stacked up, each of which isconfigured to address a particular wavelength.

Referring to FIG. 22U, two oppositely-oriented freeform optics are shownstacked in the Z-axis (i.e., they are upside down relative to eachother). With such a configuration, a compensating lens may not berequired to facilitate accurate views of the world through the assembly;in other words, rather than having a compensating lens such as in theembodiment of FIG. 22F or FIG. 22R, an additional freeform optic may beutilized, which may further assist in routing light to the eye. FIG. 22Vshows another similar configuration wherein the assembly of two freeformoptics is presented as a vertical stack.

To ensure that one surface is not interfering with another surface inthe freeform optics, one may use wavelength or polarization selectivereflector surfaces. For example, referring to FIG. 22V, red, green, andblue wavelengths in the form of 650 nm, 530 nm, and 450 nm may beinjected, as well as red, green, and blue wavelengths in the form of 620nm, 550 nm, and 470 nm; different selective reflectors may be utilizedin each of the freeform optics so that they do not interfere with eachother. In a configuration wherein polarization filtering is used for asimilar purpose, the reflection/transmission selectivity for light thatis polarized in a particular axis may be varied (i.e., the images may bepre-polarized before they are sent to each freeform waveguide, to workwith reflector selectivity).

Referring to FIGS. 22W and 22X, configurations are illustrated wherein aplurality of freeform waveguides may be utilized together in series.Referring to FIG. 22W, light may enter from the real world and bedirected sequentially through a first freeform optic (770), through anoptional lens (812) which may be configured to relay light to areflector (810) such as a DMD from a DLP system, which may be configuredto reflect the light that has been filtered on a pixel by pixel basis(i.e., an occlusion mask may be utilized to block out certain elementsof the real world, such as for darkfield perception, as described above;suitable spatial light modulators may be used which comprise DMDs, LCDs,ferroelectric LCOSs, MEMS shutter arrays, and the like, as describedabove) to another freeform optic (770) that is relaying light to the eye(28) of the user. Such a configuration may be more compact than oneusing conventional lenses for spatial light modulation.

Referring to FIG. 22X, in a scenario wherein it is very important tokeep overall thickness minimized, a configuration may be utilized thathas one surface that is highly-reflective so that it may bounce lightstraight into another compactly positioned freeform optic. In oneembodiment a selective attenuator (814) may be interposed between thetwo freeform optics (770).

Referring to FIG. 22Y, an embodiment is depicted wherein a freeformoptic (770) may comprise one aspect of a contact lens system. Aminiaturized freeform optic is shown engaged against the cornea of auser's eye (58) with a miniaturized compensator lens portion (780), akinto that described in reference to FIG. 22F. Signals may be injected intothe miniaturized freeform assembly using a tethered scanning fiberdisplay which may, for example, be coupled between the freeform opticand a tear duct area of the user, or between the freeform optic andanother head-mounted display configuration.

Various example embodiments of the invention are described herein.Reference is made to these examples in a non-limiting sense. They areprovided to illustrate more broadly applicable aspects of the invention.Various changes may be made to the invention described and equivalentsmay be substituted without departing from the true spirit and scope ofthe invention. In addition, many modifications may be made to adapt aparticular situation, material, composition of matter, process, processact(s) or step(s) to the objective(s), spirit or scope of the presentinvention. Further, as will be appreciated by those with skill in theart that each of the individual variations described and illustratedherein has discrete components and features which may be readilyseparated from or combined with the features of any of the other severalembodiments without departing from the scope or spirit of the presentinventions. All such modifications are intended to be within the scopeof claims associated with this disclosure.

The invention includes methods that may be performed using the subjectdevices. The methods may comprise the act of providing such a suitabledevice. Such provision may be performed by the end user. In other words,the “providing” act merely requires the end user obtain, access,approach, position, set-up, activate, power-up or otherwise act toprovide the requisite device in the subject method. Methods recitedherein may be carried out in any order of the recited events which islogically possible, as well as in the recited order of events.

Example aspects of the invention, together with details regardingmaterial selection and manufacture have been set forth above. As forother details of the present invention, these may be appreciated inconnection with the above-referenced patents and publications as well asgenerally known or appreciated by those with skill in the art. The samemay hold true with respect to method-based aspects of the invention interms of additional acts as commonly or logically employed.

In addition, though the invention has been described in reference toseveral examples optionally incorporating various features, theinvention is not to be limited to that which is described or indicatedas contemplated with respect to each variation of the invention. Variouschanges may be made to the invention described and equivalents (whetherrecited herein or not included for the sake of some brevity) may besubstituted without departing from the true spirit and scope of theinvention. In addition, where a range of values is provided, it isunderstood that every intervening value, between the upper and lowerlimit of that range and any other stated or intervening value in thatstated range, is encompassed within the invention.

Also, it is contemplated that any optional feature of the inventivevariations described may be set forth and claimed independently, or incombination with any one or more of the features described herein.Reference to a singular item, includes the possibility that there areplural of the same items present. More specifically, as used herein andin claims associated hereto, the singular forms “a,” “an,” “said,” and“the” include plural referents unless the specifically stated otherwise.In other words, use of the articles allow for “at least one” of thesubject item in the description above as well as claims associated withthis disclosure. It is further noted that such claims may be drafted toexclude any optional element. As such, this statement is intended toserve as antecedent basis for use of such exclusive terminology as“solely,” “only” and the like in connection with the recitation of claimelements, or use of a “negative” limitation.

Without the use of such exclusive terminology, the term “comprising” inclaims associated with this disclosure shall allow for the inclusion ofany additional element—irrespective of whether a given number ofelements are enumerated in such claims, or the addition of a featurecould be regarded as transforming the nature of an element set forth insuch claims. Except as specifically defined herein, all technical andscientific terms used herein are to be given as broad a commonlyunderstood meaning as possible while maintaining claim validity.

The breadth of the present invention is not to be limited to theexamples provided and/or the subject specification, but rather only bythe scope of claim language associated with this disclosure.

What is claimed is:
 1. A system for displaying virtual content to auser, comprising: a light projection system to project light associatedwith one or more frames of image data to a user's eyes, the lightproject system configured to project light corresponding to a pluralityof pixels associated with the image data; a processor to modulate a sizeof the plurality of pixels displayed to the user.
 2. The system of claim1, wherein the light projection system is a fiber scanning display. 3.The system of claim 2, wherein the projected light is displayed througha scanning pattern.
 4. The system of claim 1, wherein the processormodulates the size of a particular pixel based at least in part on atype of scanning pattern.
 5. The system of claim 1, wherein the size ofthe one or more pixels is modulated based at least in part on a distancebetween scan lines of the scanning pattern.
 6. The system of claim 1,wherein the size of a first pixel is different from the size of a secondpixel in the same frame.
 7. The system of claim 3, wherein the scanningpattern is a spiral pattern.
 8. The system of claim 3, wherein thescanning pattern is a toroid pattern.
 9. The system of claim 3, whereinthe size of the one or more pixels is modulated based at least in parton a pitch of the scanning pattern.
 10. The system of claim 1, whereinthe user perceives the pixels at different depths based at least in parton the modulated size of the plurality of pixels.